I think you nailed it.
Even in baseball, the players are much bulkier than in the 1960s. You see a video from then, the baseball players look like ordinary people (and you see a lot of coats and ties in the stands!).
That trend is reversing. The top young prospects are traditional baseball builds again.
Another reason the old guys looked smaller is that today they use tiny bats. In the old days, most used the heaviest bats they could swing.