According to Winston Churchill's memoirs, the Australians need not have worried. It would have taken 150,000 Japanese troops at a minimum to invade Australia, even with the jumping off points in Java just to the north. There was some considerable friction between Churchill and the Australian government because in April of 1942, the Australians pulled all their troops out of Africa and refused to help the Brits hold the line in Burma because they were so afraid of the Japanese invading their homeland.
In defense of the Australians, the British were doing a pretty lousy job running the war up to that point and the Australians just got fed up with it. Of course, the British can be somewhat excused as they had their hands full with the German Blitkrieg in their own backyard.