Lots of NE fortunes were build on the slave trade - it wasn't until all the ship owners were able to find a more profitable cargo that there was any significant support in mASS. to end the practice. The bad ECONOMICS of the slave trade were what finally killed it off. William Wilberforce (a, GASP!! radically RELIGIOUS Minister in England, the HORROR!)launched and continued the morality based movement against the slave trade for decades. Religious 'fanatics' the bete noir of all liberals launched the movement, the reality of the lousy Economics tipped over the popular support. (PLEASE NOTE: I am NOT saying the mASSES were economically sophisticated, they just recognized an opportunity to make more money with less effort in efforts which did not involve slavery. A level of perception that completely escapes today's residents up here.)
New Englanders, being good Calvinists, had (had?) a tradition of externalizing their morality, some would say of being busybodies. The culture in New England could not stand the contradiction of being a democracy ostensibly founded on the principles of equality and legalized slavery. The culture also mitigated against wholesale importation of slaves.
I don't know that there is any evidence that slave trade was ever more than an incidental part (if that) of the economy of New England.