Just under one-third of Southern families owned slaves. In South Carolina and Mississippi, almost half owned slaves.
Sometimes a person, or an institution, does something that is so horrific that it overrides all virtue the person or institution may otherwise have. John Wayne Gacy may have been a nice guy, but he murdered a couple of dozen boys. Hitler may have loved his dog, and Joan Crawford may have given money to charity when she wasn't beating her daughter with a clothes hanger.
Slavery was horrific, and if people didn't recognize that in 1791 (date of the Constitution), then they realized that over the next 70 years.
I normally would stand beside my neighbor and defend his home against federal intervention regarding how he raised his children, like if he wanted to homeschool. However, if he had his children in shackles, or abused them, or claimed to own them and be able to sell them, then at that point he lost the right to his own "sovereignity" over his family.
The South lost that right with slavery (and we can argue all day about what portion of the cause of the Civil War was slavery, because we all know there were other elements).
Even if States had not left the Union and joined the CSA, at some point the government should have forced the Southern states to abandon slavery.
I don't understand how anyone can support military action by the US against other countries in the interest of "freeing" people from oppressive governments and not understand that the concept of OWNING another human being is so contrary to the concept of basic, God-given freedom.