Based on what records? They are guessing based on an old earth mentality. "It had to be slower because everyone knows man has been around 300,000 years."
Reality is there have been periods where the population decreased. After the first 2000 years, there was this huge flood that decreased the population to 8.
Plagues, wars and famine have slowed down the growth periodically.
Even if the population only doubled every 1000 years, that still only leaves you about 100,000 years before you are in astronomical populations.
It had to be slower because hunter-gatherer technology takes a lot more real estate to support a few people than does agriculture. It basically won't support cities at all.
Only a creationist would make an argument like yours. Most people have more pride.