You're right. Even if we were to stop all immigration, we would lose the numbers game due to the birth rate.
And this is a brutal fact that many avoid even bringing up.