Historians disagree and notice contradictions and ambiguities, but they look closer in to the historical record than ideologues who make wholesale judgements based on political considerations. There is a great mass of historical writing on unionists in the Confederate states. Maybe you don't have time to read it, but if you don't, you shouldn't make claims that a little reading might disprove. If you are really interested, you can start here and move on to the works listed in the bibliography. But it helps if you don't read in order to tally points for this or that side, but to try to understand complex realities.
The initial Virginia, North Carolina and Arkansas conventions voted against secession. Tennessee voted against having a convention. Clearly signs of not negligible unionist sentiments. After Sumter sectional passions carried the day, but even then, close to 1 in 3 Tennesseans voted against secession. The collapse of unionist sentiment in middle Tennessee didn't affect union strongholds in the eastern highlands of the state.
North Carolina had no referendum, but before Sumter unionism was particularly strong in the western part of the state. There was less disaffection in Western North Carolina in 1860 than in other Southern highland regions, and consequently more support for secession after Sumter than in the neighboring part of Tennessee. But as the war continued, support for the Confederacy crumbled in large areas, and older unionist loyalties reasserted themselves. There was much desertion and disobedience, and some guerilla activity. And in Western North Carolina, as in East Tennessee, distrust of Democrats and the myths of the Confederacy lasted long after the Civil War.
Virginia was a special case. It has been said that secession narrowly carried the day in the Western part of the state, but there have also been allegations of vote tampering, fraud, intimidation, and the destruction of records. It may also be that some Unionists boycotted the election. I don't know what happened, but the state convention voted against authorizing a secession referendum on April 4 by a vote of 89 to 45 and then decided for a secession referendum on April 17 by a vote of 88 to 55, an indication of a large bloc of unionist sentiment that wasn't reflected in the referendum counts. While there was much secessionist sentiment in the southern and eastern parts of what's now West Virginia, support for the union was overwhelming in the northwest near Ohio and Pennsylvania. It's hard to believe that voters who sent anti-secession delegates to the convention would somehow have been converted wholesale to pro-secession views.
Virginia was a special case because unionists took action and formed a government with support of the US government, rather than simply suffer under Confederate rule. This may have been a questionable or dubious procedure, but the same could be said of all the secession movements of that time. If West Virginian leaders were rash and heavy-handed, they were acting in the spirit of their scessionist contemporaries, who likewise wanted to seize the day and drag the recalcitrant or reluctant along after them.
Even the Deep South wasn't unanimously in favor of secession. The Alabama convention favored secession by a vote of 61-39, a nice margin in an ordinary election, but hardly a ringing endorsement of a major constitutional change. A sizeable part of the state was against secession and never gave much support to the Confederacy.
Georgia also had strong unionist sentiment, particularly in the North of the state. The vote to hold a convention (50,243 to 37,123) was no blow-out. Not everyone who voted for the convention necessarily favored secession, and the vote totals in some counties were questioned. In 1972, the Georgia Historical Society reexamined the results and found a narrow majority against immediate secession. Many of those votes for "cooperation" and against "immediate action" likely weren't unionists, but the vote count that had long been cited as support for secession didn't reflect Georgia sentiments. Prosecession delegates didn't even run in some North Georgia counties and weren't elected in others.
The convention's vote 208-89 was more one-sided, but the final decision to seceed was not submitted to the electorate. After the war Alexander Stephens said that if it had been, secession wouldn't have passed. Under the circumstances, his testimony may not be the most reliable, but given the way that Georgia's voting systems distorted election results and frustrated majorities, he may have had a point, that dissenting opinions might not have been fully reflected at the convention. During the war North Georgia, like Western North Carolina and East Tennesee saw peace candidates, draft evasion, desertion, fighting between home guards and deserters and local militias and reprisals by regular Confederate troops.
In some Deep South states sentiment was more strongly in favor of secession, but we can't say for sure if they didn't hold referenda. When the shooting had already started, the passions of the moment were apt to push referenda in the Upper South towards secession, and to override earlier qualms. But pro-union sentiments have to be taken into account in five or six confederate states. By the end of the war, all of them except South Carolina had mustered units in the US Army. Given how the war ended, that's not the best indication of sentiments, but one can't ignore or deny the strength of unionist sentiment in some parts of the Confederacy. If Davis gambled that war would solidify support for secession and the CSA he was right in the short-run, but wrong over time. Those converted from mildly unionist to mildly secessionist sentiments didn't always stay the course with the Confederate regime.
Does it matter today? It does.
First of all, it's an indication of the failure of ideas of unilateral secession. Rely on a single vote taken at a time of duress to radically change people's citizenship and status in the world and you invite war, since a referendum a few weeks before or a few months after would give a different result. Stampede people into taking a given political course at a time of crisis, and you'll pay for it later, when people have second thoughts. Somebody always gets dragged along in such secessions against their will. That's why a more deliberative and mutual process is necessary. It allows dissenters to work out their own future.
Secondly, it's wrong to exploit people's love of liberty or distrust of outsiders to promote a cause that their ancestors may well have fought against on the grounds that it offered them neither greater individual freedom nor greater group autonomy. Some of those who've adopted the Confederate Battle Flag as their symbol ought to at least examine the possibility that their own ancestors might have stood by and fought for the Union with pride. It would introduce nuance, complexity, and ambiguity, to what too often is a tribal, "them against us" argument.
To be sure, there were many convinced secessionists in regions with many Union supporters, but it's arguable that the better part of the population -- those who were more patriotic, more devoted to justice, and less swayed by appeals to emotion and privilege -- went with the Union. Some of those in the Old South who were in real and practical ways the most libertarian and anti-statist supported the Union, recognizing greater oppression in state governments and breakaway confederacies, than in the distant national government in Washington. That may not be the case today, but it is something that ought to be acknowledged.
As for the tariff, I followed Alexander Stephens and others in saying that Southerners could have written the tariffs they wanted. This was an exaggeration, as any tariff would have to reflect the different interests of different parts of the country. But the basic idea was sound. The high tariffs of 1828 and 1832 were revised downward in 1833. An upward revision in 1842 was corrected downwards in 1846. The downward movement in 1857 would inevitably be revised upwards. And that upwards movement would eventually have been revised downwards had everyone kept their heads about slavery and secession. Such were ways of ordinary politics in 19th century America. The fact that extreme free traders didn't get their way in 1842 or extreme protectionists win out in 1846 was no cause to dispair or rage at the state of American politics. There was nothing apocalyptic about tariff legislation. The same could not be said for the slavery issue.
As for Calhoun, he did see slavery as a "solution" to the South's racial problems, but he went much farther than that. Confronted with the European revolutions of his day, he came to see slavery as a safeguard against the evils of revolution, socialism, and democracy. In this mood, Calhoun saw the racial split between master and slave castes as a positive bulwark against the dangerous tendencies of democracy. There was no wall of separation between Calhoun and other thinkers who built upon his belief that slavery was a "positive good."
There was an evolution at work in Calhoun's thinking and that of his contemporaries, but it ran in against what one takes to be the path of progress: In 1823, Calhoun said that slavery was "scaffolding, scaffolding, Sir, it will come down when the building is finished." By 1838, he asserted: "Many in the South once believed that it was a moral and political evil; that folly and delusion are gone; we see it now in its true light, and regard it as the most safe and stable basis for free institutions in the world." Later in life he was saying that "Slavery is indispensible to a republican government. ... There cannot be a durable republican government without slavery."
There were contradictions in Calhoun's thinking. For all his reputation as a rigorous logician, he expressed quite different views on different occasions. In this he was a politician, lauding slavery as the basis of freedom to wealthy Southern audiences, while denying it in letters to and conversations with Northerners. He prefered to restrict himself to defenses of Negro slavery in current conditions, but couldn't resist making broader claims for slavery as an institution, outside of the existing racial context.
It's highly unlikely that Calhoun wanted to enslave free Northern White workers, but it's clear that he was dubious about their value to the polity. One would have little trouble constructing a defense of slavery as an institution and of racial divisions between rich and poor castes based on some of Calhoun's statements, and some Southern intellectuals did just that.
And as regards Toombs you might look at his speech before the Georgia Legislature on secession or his speech on resigning from the Senate. These are only excerpts, but they indicate that he had more on his mind than tariffs.
Was he making "libertarian" arguments? Is Toombs's defense of the rights of slaveowners "libertarian"? Is his attack on abolitionists? Or his proclamation that he should be able to take his slave with him anywhere? Are fears for the survival of slavery libertarian? One could go further. Are the complaints of ill-treatment and grievances of one section necessarily libertarian? Or its defense or promotion of its own interests? Or its dreams of future wealth and grandeur? Looking secession as a "libertarian" solution can make some take all expressions of support for secession as "libertarian," though they may be anything but that.
In general, the idea that the secessionists were "imperfectly libertarian, but libertarian nonetheless" is a weak one. This is America, and all of our political disputes tend to be cast in terms of liberty and rights, just as a wise lawyer clothes his client's interests in "rights talk" whether or not they are really justified. So all American politicians are more or less "imperfectly libertarian," even Abraham Lincoln. Secession is regarded by some as a libertarian idea, an anti-statist concept, consequently secessionists are given haloes by some libertarians. But this is a dubious, question-begging proceeding.
First, not everyone who advocates breaking away from a larger political unit is justified or truly libertarian, no more than everyone who supports the overthrow of an existing government is justifed or a true lover of freedom. An oversimplified version of libertarianism leaves obligations and procedures out of the picture and produces perverse results. Anarchy would be the result of allowing any group exit from larger polities at its own wish, and most people generally fear anarchy. But then anarchy is not a bad word or thing to some radical libertarians. Actions that most people would characterize as reckless, irresponsible, and dangerous, are celebrated by them if such actions weaken the state.
Second, secession is a means that has little to do with one's ends. If one group desires independence in order to secure, extend or perfect its oppression or exploitation of another group, libertarians, if they care about liberty, ought to think twice about supporting the independence bid. That's not to say that they should justify or join in crushing it, but they shouldn't give it their approval. To blur or ignore important questions of means and ends serves no useful purpose. Describing the rebels of 1860 as "imperfectly libertarian" makes a hash of libertarianism and a mess of politics in general.
Third, There's a hypocrisy in the libertarian coloring some give to state's rights theories of secession. If one is to support Southern slaveholders desire for self-determination, one has to grant similar support to the desires for liberty and autonomy of free non-slaveholders, and of slaves. A narrow focus on the biggest and possibly least justified revolt slights other people's efforts at freedom. When Rockwellites talk as much about the slaves' struggle for freedom or the plight of Southern highlanders dragooned into fighting for an alien Confederacy as they do about the Confederacy as victim of the evil Northerners, one may be able to take their claims to be libertarian more seriously.
Fourth, applying a late 20th or early 21st century ideology to mid-19th century politics is a dubious procedure. It means ripping ideas out of their 19th century context and substituting one's own context for that of a previous generation. If it's to be done at all it has to be done fairly and evenhandedly. To disregard all questions of slavery, focus on tariffs, and annoint the Confederacy as "imperfectly libertarian" without irony, would be a mistake. The idea that somehow one need only preface one's comments with "Of course, slavery was wrong," and exclude it from further consideration is ridiculous.