The glacial periods can last thousands and thousands of years - for people from those times, it's all they would have known.
The theory of course is that the coastal people were the advanced ones and those inland pretty much primitives. The end comes and wipes out the coastal civilization that rose up and all that survive are a few refugees from the coast and the move in with the primitives and the primitives memorialize this in our mythology of their ancestors meeting gods or heroes. When put that way it's a plausible scenario.