ACtually, it does, to some degree. Immigrants tend to have more children than natives. Had we traded 40 million babies (and that's assuming that there is no deferment effect) for 40 million immigrants, our poulation would be 10-15 million less.
The biggest factor in our population growth isn't immigration at all . . . it's the fact that we're living longer than ever before. The fastest-growing age cohorts in this country are actually the elderly and even the frail (90+ years old) elderly.