The vast majority of the US population was rural in 1860. The world that most Americans knew in 1860 was farming and farm life. If you'd heard of what was going on in Manchester or Lowell, or even if you'd been there, it was likely to bewilder and horrify you.
You might not want all that noise and smoke and squalor coming to your town. You might not want to lose control of your society to upstart manufacturers or urban mobs. And you might not think it was truly the wave of the future. If you could benefit materially from the growth of industry without importing all of its problems you probably would try to.
Look at how agricultural elites responded to industrialization in other countries, Britain, Prussia, Russia. Those countries did industrialize but the existing landowning elites tried to keep a firm hand on political power. Political and military careers maintained high status to keep mere manufacturers from getting too much power.
And in colonies and out of the way regions, landowners were all too willing to make money supplying industry and the cities with raw materials and food without going industrial and urban themselves. Ireland didn't want to become Britain. Nor did Australia, or Canada. They could make money supplying Britain with wool or grain or timber and remain emptier, freer, more egalitarian, less driven and troubled societies.
The few Southerners who believed industrialization was the way ahead tended to be in favor of tariffs and national banks and public works projects and government subsidies and even government-owned enterprises. They were the heirs of Hamilton and the Whigs.
If you spent years complaining about such policies, as most Southern politicians (who saw themselves as the heirs to Jefferson) did, you weren't going to support such policies. Because of your agrarian, Jeffersonian values, or because you wanted planters or farmers to hold on to the power they had, or because you opposed such developmentalist policies, you weren't likely to be keen on industrializing the South.
Today, society is far better informed and more economic minded and entrepreneurial. We could see that Silicon Valley, say, was the wave of the future. We can also see how that turned out. If you could supply Silicon Valley or invest in it without bringing all its problems to your own home town, wouldn't you do so? Back in the 1850s and 1860s when things were much less clear, wouldn't some of the few people who knew what we now know think the same way? Why build a smoky industrial city with poor housing and drainage and an unruly working class when you could just make a profit supplying the city's material needs?