Actually a couple of centuries of population decline still matter even if the overall timeline is 100,000 or more years. If you’re positing at 1% annual growth then 1 year of -1% growth (decline) is taking 2 years out of your graph, the year of decline and the year to make it back. If your decline gets larger the effect on your graph gets worse. 1 century of 2% decline just cost your graph 300 years. That doesn’t even get into things like known sharp contractions like the Black Death, that chopped off 1/4 of the world’s population in 2 years. One or two of those will tear your graph to pieces, and there have been dozens.
Sure, the sharp declines from extraordinary events will have a significant effect, but I’m still not convinced about minor declines, even for centuries, being very significant. Firstly, most of them are local, not global, and they may well be compensated for by increased growth in other areas when you look at the global scale. Second, even if you have a century of decline, and it takes two centuries to make up for that, you are talking about a fraction of a percent of your timescale. That’s not going to be more that the slightest hiccup in your trend line.