The only reason we need to keep growing the population in the US is because we have set up the social security pyramid scheme. Without an ever-increasing taxpayer base, SS falls apart.
Other than paying for the welfare state, there is absolutely no reason that we need anything more than replacement rate fertility in this country.
Replacement rate is a nice theory, but it doesn't work in practice. How many "free will" replacement rate societies exist around the world? Societies are either expansive and limited by their environment/conditions/emigration or they are collapsing and supported by immigration.