Clearly had the vast majority opposed slavery, the southern concentration of slave holders would have been out of luck. But either rights and morality are fixed and permenant, or they are will'o the whims. You can't have it both ways. If rights come and go with the wind, then you slave holders have no complaints if someone came along and gave you a good whipping.
at the time the founding fathers wrote the Constitution, these "minorities" were considered 3/5 of a person, and had little or no rights
Slaves were denied freedoms, of course. The compromise in the Constitution was opposed by the south. They wanted slaves counted as full persons so as to get more representation in the congress, but not to give the slaves a vote. Very cynical. In unfortunately agreeing to tolerate slavery, the free states pressed to count the non-voting slaves as less so as to avoid giving the slave states additional congressional power.