The Founding Fathers had the right balance. Leave the individual to self-govern and only what he cannot reasonably be expected to do himself give it to the next level government.
All political positions have their limits. The Founding Fathers may have wanted to allow people into the country freely, but they denied them the vote until they became citizens.
There was very little immigration to America for the first couple of generations and Catholics were almost non-existent until they were allowed to start coming in by the politics of the 1840s.
For fun try and find a source that lists immigration for say 1790 and 1800 or even 1810, they used to be available but I can’t find them now.