But they wouldn't. That's to say, there isn't much talk of secession where individual liberties or the size and power of government alone are concerned. There has to be either 1) a threat to some state's (or group of states') economic interests or 2) some powerful moral issue (or both 1 and 2).
Political divide and conquer would no longer be as practicable by Federal politicians.
What you'd have is political blackmail by various state elites which threaten to withdraw if they get their own way. But that doesn't preclude political divide and conquer. The federal government buys off those powerful state elites, and then it's free to do as it wishes.
If we were free to leave it is not unlikely that we would not peacefully reassemble in time, under a new Federation with stronger limitations.
So if slave-owners broke up the union because they felt their interests weren't being defended. If they walked off with what they wanted and left the rest of the states to fend for themselves, we'd have said, "Come back! We'll give you all you wanted and more!" Not very likely. And it's also not very likely that -- having gotten their hands on a government of their own -- those slave-owners would want to put themselves under a government that they might not control.