Of course, Marse Bob, you are correct. However, the spiritual impetus behind Abolition justified everything in the Yankee mind.
There were many southerners who were against slavery, too. Most on moral grounds, and a perceptive few on economic grounds.
Just a reminder. There were thousands of blacks who were slave owners.