Families who own slaves is one reason I spoke above of "households". "Though it's undeniable that the Confederacy gave special status to slavery, records show that most Confederate soldiers came from households that didn't own slaves." States varied in how many families had slaves, but they were in the minority in every state. Wikipedia:
"One estimate is that in 1860, about 25% of households and 5% of the population (384,000 people) in the South owned at least one slave. An alternative estimate is that 36% of men lived in slaveholding families, and the percentage of men who had economic ties to slavery was much higher."
Still most didn't.
But still it's easier to understand how the South could rebel when 25% of its people derived direct benefit from slavery and countless more derived indirect benefit.