"Every society has a right to fix the fundamental principles of its association, and to say to all individuals, that if they contemplate pursuits beyond the limits of these principles and involving dangers which the society chooses to avoid, they must go somewhere else for their exercise; that we want no citizens, and still less ephemeral and pseudo-citizens, on such terms. We may exclude them from our territory, as we do persons infected with disease." -- Thomas Jefferson
Jefferson was wrong and should have used the word power instead of right. Of course, Jefferson was wrong to own slaves too. "Runaway now little man", the peasant said to the king.
Jefferson wasn't a simpleton.