It’s more than that. Farmers didn’t have the tough life of a hunter-gatherer.
They were softies.
The population rose when people settled down to farm. It meant more disease and grinding labor, so farmers didn’t have life easier than hunter-gatherers. When there were fewer people around, there was less competition for berries and game.