As it was explained to me once in a geography or history class long ago, the presence of plentiful hydro power in New England was decisive in starting the industry there. Other Fall Line towns in the Piedmont had mills, too, but not as big as New England.
The presence of the Great Lakes and the Erie Canal was decisive for Northern versus Southern locations. Plus, the big ports were in the north since late colonial times. New York ran away from the rest of the country after the Erie Canal was opened.
Sure, there was coal in the South, but it was the combination: proximity to water transport AND raw materials AND population centers AND ports AND finance AND markets.
It all added up in favor of the Northeast. It wasn't "cultural" -- that's a political construction, as I said before.
I'm not aware of any significant textile mills operating in the Piedmount before the 1890s, but I could be wrong. But by 1860, mills were no longer relying on water wheels. They were steam powered meaning they could be build anywhere that there was access to fuel, and the south had plenty of that available.
A little long, but an interesting read.