Northerners didn't own slaves directly....not by the mid 19th century. They sold slaves then profiteered off of the goods their labor produced. As the 3 New England Journalists who wrote Complicity put it, it was slavery the way the North liked it: "most of the profits and none of the screams."
There you go… it was all the Damn Yankees fault. Those poor Southern planters had no part in those unfortunate problems. It was those Damn Yankees!