To regulate Commerce with foreign Nations, and among the several States, and with the Indian Tribes;
To establish an uniform Rule of Naturalization, and uniform Laws on the subject of Bankruptcies throughout the United States;
To provide for calling forth the Militia to execute the Laws of the Union, suppress Insurrections and repel Invasions;
From the Constitution. Congress has always had the power to regulate immigration. It has not needed to grow bigger to do so.
In your world are there any borders? If not, why not?
Yes, it has always had the power, but rarely acted on that power, until this century (although it did restrict immigration of various groups, depending on the political whims of the populace, chinese comes to mind). Government has gotten bigger to regulate all of this and as the regulations increase, so does the size of government. The US government used to pay for advertisements in Europe to try to get more immigrants to come here. Europeans governmetns were upset because many of their citizens were leaving the freedom suppressing countries of europe. Of course, I don't suggest government steal any more of our tax money to enact such a a progmam...
I don't propose open borders as terrorists need to be stopped. I'm in favor of moving the military towards the borders and building a fence. However, I'm also in favor of letting anyone who can get here on their own free will into the country.