You are very wrong here. The slave owners were aristocrats who's lifestyles were only slightly threatened by extinquishing slavery:
"Slavery is likely to be abolished by the war power and this I and my friends are in favor of, for slavery is but the owning of labor and carries with it the care of the laborers, while the European plan, led on by England, is that capital shall control labor by controlling wages. The great debt that capitalists will see to it is made out of the war must be used as a means to control the volume of money."
Private circular of Northern banker, late 1861.
The ones of which you speak were not slave owners. The descendants of slave owners were not those who hung civil rights activists in Missippi. They were the rank and file rednecks who had to feel like they were better than something and they could only single out blacks. Turns out they were not better than anything, certainly not better than the black people I have known who have worked for my mother in my house, worked for me in financial services, and whom I have worked for in bank auditing.