In 1980 Carter got the bright idea that not going to the Moscow games would somehow get the Soviets out of Afghanistan, what it actually did was destroy lifelong dreams of many American athletes.
In 1984 the Soviets boycotted the LA games, there were some bright spots but it was still impossible to shake the feeling that any victory was a little hollow if the commies weren't competing.
By 1988 was okay but nothing really stood out and by 1992 the pretense of amateur athletics was abandoned completely with the US "Dream Team" (which was a great basketball team, I just don't think it had anything to do with the spirit of the Olympics).
Even 76 had the boycott by African countries due to New Zealand being allowed in after their rugby team toured South Africa.
I attended the 76 games in Montreal. I wouldn’t rate them as great. There was an African nation boycott over New Zealand playing a non olympic sport with South Africa and a controversy over Taiwan calling themselves the Republic of China. It was also the first summer games with a very heavy security presence since it followed the 72 massacre of Israeli athletes. There were some great athletic performances with Nadia and Bruce Jenner and others but I found the overall atmosphere there that year dismal. The hosts became the first ever to not win a gold medal. East Germany won a ridiculous number of medals which are all discredited now but stand because the IOC decided it was too messy to strip so many, years after the event. There was plenty of Olympic spirit sucked out of the games that year.
You may be onto something there, additionally they split the summer and winter olympics into two different schedules for ratings/$$$.
How can you miss it when it never goes away?