Or maybe the weak and infirm stopped STARVING?
Quite possibly.
However, if you keep the weak and infirm alive, the average health of the population goes down.
That too. In a hunter/gatherer culture, you have to be fit enough to move with the tribe as it follows the herds.
One advantage with agriculture is it allows a square mile of land to support more people than hunting does. Having more men makes it easier to defend your territory, even if the average farmer is less physically fit than a hunter.