Except that they all have predators. They have limited breeding grounds, they are killed by changes in weather, etc. Man with tools is the top of the food chain except for disease and parasites.
Sure without agriculture man would be limited by the food supply and periodic famine. But the earth is still capable of supporting a lot of men without agriculture, and we just don't see the record of lot of men having been around for 300,000 years.
Two things. We are not the top of the food chain. Ask anyone who's come up against a shark, tiger or bear. Secondly, unlike other species, we tend to kill others of our kind.
And, don't discount diseases, famine or parasites. The Black Death claimed a third of the European population in the 1300s and a 19th century famine in China claimed millions.
Hell, people are one of the biggest hindrances to population growth. Even if you discount the 100 million or so who perished in the last century, you've still got the millions slaughtered by the Huns, the Mongols, the Saracens, the Crusaders. Then you have the tens of millions of Native Americans who didn't survive contact with the Europeans and their diseases.
Like I said, don't let reality interfere with your calculations.