Rod Serling understood in 1960, and did a Twilight Zone episode about it:
The Monsters Are Due on Maple Street
I remember that one.