Blame a tiny handful of wealthy men in Charleston. Most southerners didn’t own slaves, lower income whites actually resented slave
labor because it enabled the wealthy to avoid hiring workers they would have to pay, and nobody - except for the Middletons, the Moultries, the Pinkneys and their ilk - thought this would be a good idea in any way.
This is why most northerners hated slavery too. We are led to believe that Northerners hated slavery because of concern for black people, but this is as far from the truth as it is possible to get. They hated slavery for two main reasons.
1. It took away wages for people who would be paid to perform work. People who trade their labor for money saw slavery as a threat to their livelihood.
2. Northern people absolutely hated blacks, and did not want them in their communities. Slavery caused blacks to associate with whites, and the very prejudiced whites wanted none of that.
3. A third and lesser reason is they hated the rich slave owner getting rich without having to work for it. This was an envy driven hatred.