Anyone who has ever gone hunting knows that it would be a rigorous lifestyle if you had to live off of your kill.
If you could live off of "the one that got away," we'd never would've needed to invent farming :)
Back when the earth's population was low, there were more animals around. You should read what the Europeans who discovered North America say about the profusion of wild game and fish.
Also, hunter-gatherer types are pooped out old grandfathers at death's door by the time they reach 30. All that excess energy kids have? Among hunter-gatherers, they use it to get food.