Excellent point. It should be noted, however, that southerners believed that the only way they could protect their way of life and the "peculiar institution" on which it was based was to maintain that dominance.
They believed it was rule or ruin for them. And they were quite correct in this belief.
The irony, of course, is that their more and more desperate attempts to protect and spread slavery just drove away their natural allies in the West and as a result created the northern coalition that eventually destroyed them.
Part of the problem was that slaves had to be kept ignorant to be slaves.
To be ignorant, was to have low productivity. Agriculture had to be dumbed down so that ignorant slaves could accomplish simple tasks.
Any slave that acquired a knowledge of simple geography could decide to no longer be a slave. Many did. By obtaining papers from Canada, they secured protection from the professional kidnappers that were the hirelings of the slave power.