I've been watching the second season again and there's an episode where the shattering noise and blinding white light occur again exactly as when the island moved in the fourth season.
When I watched the show the first time I didn't understand what that meant. But it seems like the island moved late in the second season, and perhaps that's what permitted all the time-traveling of people like Jack's dad.
Was that when Desmond turned the key? I wonder if the Swan station's real purpose was to keep the island from moving? Possibly, the 108 minute dead-man switch was to make sure if someone found the island and disrupted the residents, that it would move (through time?) and keep the island safely hidden.