It's said the Irish settled much of the South, especially the Protestant Scots-Irish from Ulster. They had a reputation as pioneers and as brawlers. The other group was the Borderers who lived in Northern Britain and Southern Scotland. Like Northern Ireland it was a comparatively wild and remote part of Britain in the 18th century. The two groups blended together on the American frontier.
Probably it wasn't so much that such people were different from the English. You could see a lot of the same characteristics in Dickens' London or in early Australia. But the Ulstermen and the Borderers were further away from cities and the very settled and controlled life of East Anglia, where many New England Puritans came from.
David Hackett Fischer wrote a lot about this topic in his book Albion's Seed. It's worth a look. One thing he says is that the Gaelic-speaking Highlanders who settled in the Cape Fear area didn't mingle well with the Lowland Scots or the Scots-Irish. They had been on opposite sides of some conflicts in the old country, and didn't trust each other. The Highlanders had had their rebellions and been crushed mercilessly. Thus they tended to be more obedient and law abiding in the New World.
Great book - Albion's Seed.
Peter Wood argues in "Black Majority" that the slaves in South Carolina were better able to maintain their culture because they had less interaction with whites. Of course, that culture in and of itself was a blend of the various african and carribean locales of the slaves.
In the highlands of Appalachia, there was more interaction between whites and blacks, in part because there were far fewer slaves and in part because there were more poor whites. The poor economy meant that slaves and their owners often lived in tight quarters and poor whites and free blacks often lived side by side.
Long winded post to say you are right!