The difference between the North and South of that period was that even if the Northerner did not love the black man, he did not think he had a right to live of of his labor.
Then why did New Yorkers (the last I checked this was a very northern state) riot when Lincoln (the racist) issue the Emancipation Proclamation?