In general, those states which had the highest proportions of slaves and slaveowners voted for secession, and voted for it early. Those with lower percentages went along later or, where the percentage was lowest, not at all.
This held for the 1860 elections, if a vote for Breckenridge can be taken as a vote for secession and one for Bell as a vote against it: Breckenridge tended to carry the states with more slaves, Bell, those with fewer. The exceptions were Delaware and Maryland, where Bell and Douglas split the moderate vote, but they don't affect the general rule.
It is true that in some states the large slaveowners of many were Whigs and later Constitutional Unionists. They thought that there was little to be gained by agitating for the expansion of slavery, and there was still some antipathy between Jacksonians and the wealthy. A very large slaveowner might be more immune from popular agitation, and might support a more moderate course of action than his neighbors.
Party politics played a role, as Democrats came to lead the secessionist movement and others held back. Geography was also a factor: the largest landowners of the Mississippi Delta tended to support the Whigs, and the rest of the state tended to go Democratic, but the leaders of both parties were most often large landowners or connected to landowning families. Then, as now, one has to be skeptical of Democratic claims to be the "party of the common man," since the leaders of the party very often were anything but that.
South Carolina, the state that sparked secession and war, didn't really have a working party system. The same wealthy families tended to divide offices among themselves. While party differences might have blunted drives to secession elsewhere, their absence in South Carolina encouraged secessionist tendencies. In the state where the planter elite were most in control, secession went furthest and fastest, and that's no accident.
What you're seeing is a preference for direct democracy, expressed in plebicites and referenda, over representative democracy. One problem with direct democracy is that it tends to appeal more to the passions than to "rational, problem-solving" behavior. It can cause a lot of trouble that can't be undone. Another difficulty is that plebicites can be manipulated. It's alleged now that a Georgia vote taken as a mandate for secession was actually won by the other side.
And it's not at all clear that the idea of breaking away to form communities of like-minded people actually promotes individual liberty. Indeed, it may be a very bad thing for the rights of the persons when factions determined to have their own way in all things form their own breakaway nations.
Representative government has a way of bringing people of different opinions together to work out a way to live with each other. Where secession is enshrined as the main principle of government, people move to extremes, tear countries apart, and pass laws that are often to be more offensive and repressive than is otherwise the case.
It polarized the nation even until now. I know it did at the level of the typical American. The ordinary southern soldier farmer had more in common with with the ordinary Union soldier farmer from Indiana that he was fighting than he did with the slave owner planter whose interest he was fighting for. The passions of the time produced unnatural enemies and unnatural alliances.
It didn't do all that much for individual freedom, when a faction determined to have its own way in all things won the White House in 1860 and proceeded to bulldoze the rest of the country.
In that case, bailing out was the intelligent alternative to staying, and being dunned and swindled.