It's not really a concern, either way. As of 1999, the American fertility rate was 2.08 children per women, which is just a whisker below replacement levels.
There are indications that the American fertility rate is in the process of increasing slightly.
So, we're basically at replacement levels, maybe a bit under. We can certainly make up the shortfall, if any, with smart immigration. Currently, our high immigration rates mean that the American population is growing at above replacement levels. So, we could cut immigration substantially and still be at replacement level.
Other than paying for welfare programs, there is no reason to expand the American population.
Ever seen "Scrooge"?