So it's always best to look to what the majority thinks when it comes to immorality and not a higher authority? If majority decides something is not immoral, then it's not immoral?
...and YES, at the time the founding fathers wrote the Constitution, these "minorities" were considered 3/5 of a person, and had little or no rights, hence the answer to your question is NO...
Thank goodness for Lincoln and the Radical Republicans then, they fixed that situation.