Hmmm... a quick reflection on history will reveal that, while the southern colonies used the labor, it was your northeastern colonies that did the importing of it...
Can you name me one ship that sailed from NH as a slave trader?
A fact of family history. My dad was from upstate NY, and his family were among the earliest settlers in MA (1630). My mom's family was/is from central Louisiana by way of Georgia of similar "early arrival date"(and yes, I'm aware of what that implies). We'd go up to NY every year to visit, and my Yankee aunt was very proud of her great-grandpa's service in the "War of Northern Aggression"/"Civil War". Had his sword and stuff.
Studying the family history, I come to find out that the first MA progenitor was also the purchaser of one of the first lot of slaves brought into MA.
My mom's family were small farmers, and AFAIK, never owned slaves.
I thought that was somewhat ironic.