The organising principle of Southern society was the natural superiority of the white race over the black race. Nothing was going to interfere with that. If slavery was either outlawed or phased out that meant at the least the former slaves could start to outvote the white populace and who knows where that may have lead. At the most it would mean that whites and black were morally equal. Both outcomes were completely unacceptable to the white populace whether slave owner or not.
Slavery was immoral true but slavery was legal in ALL states North and South in the 18th, early 19th century. Racism didn't end above the Mason-Dixon, Lincoln himself was a racist to the nth degree. That is fact.