Nothing in the constitution limits individuals. The constitution limits the government. Every citizen is allowed to determine his own criteria for choosing for whom to vote.
If only Carson’s critics understood such.