The North did make more profit servicing the goods produced at least in significant part, by slave labor than the South did. Obviously most Northerners were not engaged in Shipping, Banking, Insurance and Wholesaling for those goods. A significant number, but nowhere near most. So I stand by the statement that most Northerners didn't think much about slavery in their daily lives.
Really. Show us the numbers. What percentage of the Cotton crop did they get. How many “Northerners” made more profit than the plantation owners.