"Every society has a right to fix the fundamental principles of its association, and to say to all individuals, that if they contemplate pursuits beyond the limits of these principles and involving dangers which the society chooses to avoid, they must go somewhere else for their exercise; that we want no citizens, and still less ephemeral and pseudo-citizens, on such terms. We may exclude them from our territory, as we do persons infected with disease." -- Thomas Jefferson
Jefferson was wrong and should have used the word power instead of right. Of course, Jefferson was wrong to own slaves too. "Runaway now little man", the peasant said to the king.
The quote is taken out of the context of the total concept of govm't held by Jefferson. Using the quote this way could also be done by those wishing to validate the imposition of socialism, communism, theocracy, or some other authoritarian form of rule. Jefferson held authoritarian rule to be abhorrent, thus using his name to justify something other than what he promoted is just as abhorrent.
Jefferson didn't write the Constitution, which only mentions rights in conjunction with people. Note also that Jefferson says "society" not "the government" or "the state", they aren't the same thing. Society, IOW the people, may of course grant the State the power to enforce such principles of association, through some Constitutional provision or via Congress by a law not violative of the Constitution.