Sad thing, as you probably know,is that only 10% of Southerners owned slaves to begin with. I think there was enough blame to go around on both sides.
At that time the South was undereducated and many people were susceptible to the Lies of the slavers which transformed the fight for their slaves into a fight against "invaders".
Particularly revolting was the exemption from military service of large slave owners while the poor whites were drafted to fight to protect those very owners. This injustice and irrationality of the southern system and rulers reminds me most of Czarist Russia in its corruption of and destruction of the human spirit and blind fight against freedom.