Modern Americans have summer lodges for vacation and some have hunting cabins in other states; we don't need to reside in them or even near them year round. So why would you expect ancient cultures to have spent most of their time with any structure they built?
Modern structures are mostly temporary, being made of wood and other non-permanent materials. They go up fast and last a couple of generations. This structure was stone, took tremendous craftsmanship and certainly many years to build, perhaps even generations. That is why I would expect it to be the centerpiece of a civilisation, rather than a roadside attraction for a bunch of nomands as they wander by every once in a while.