The main problem is host cities are required to upgrade public infrastructure, build new stadiums and facilities and undertake a massive amount of preparatory and organizational work.
All of that is expensive and few host cities will make money off the Olympics. The IOC has priced the Games out of the marketplace.
I would think if they had a permanent site, only minor improvements would be needed every time out. Yes, I think Greece should be the permanent home of the Olympic Games.
I further realize that bilking the taxpayers has been part of the game behind the Games for decades.