You're assigning today's environment to that of 100,000 years ago. When you're not the highest predator on the food chain, bad things happen and the average age is significantly less. I have no doubt that early man existed and I think carbon dating proves it. Why growth was slow can probably be explained by environmental conditions, high infant mortality, and a bazillion other factors.
“high infant mortality”
A high adult mortality rate as well.