If that were the case, then the secession movement wouldn't have garnered the thundering majorities that it did in the States that subjected their secession proclamations to plebiscites. In Texas and Virginia -- in Virginia you think you'd have had more "dissent", if that is what you are looking for -- the margins were of the order of 3:1 - 4:1.
Voting to remain in the Union, effectively, were the biggest Southern planters who feared a war, and who nominated and touted the Constitutional Union Party's candidate, John Bell. They got about 20% of the vote in Texas, in the 1860 general election. Lincoln got zero, zippo -- and Judge Douglas got very few.
Secession carried the day because that was what the People wanted. Legislatures and moneyed interests were a lot more cautious. Virginia, Tennessee, Missouri, North Carolina, and Kentucky all played a waiting game or tried to remain "neutral" like Missouri and Kentucky. Lincoln wasn't having any -- he mounted a putsch, a coup d' etat, against the State of Missouri using Union sympathizers organized illegally as "Home Guards" from the Wide Awakes and German immigrants, and they marched out and took over the State of Missouri, starting with the Missouri Militia, which they confronted in arms and forced to surrender (the Militia had no idea they were coming for hostile purposes and were caught utterly by surprise). Then they chased the state administration and legislature from their posts and took over the State by coup de main, all at Lincoln's orders, conveyed by his political cutout, Frank Blair, and Captain Nathaniel Lyon who, while a serving officer of the United States Army, before Lincoln's inauguration, had armed the Wide Awakes with weapons smuggled over the river by the governor of Illinois, from Illinois Militia armories. After the inauguration, he armed more companies using the absence of a superior officer to open the United States armory in St. Louis.
So far from an "oligarchy" demanding secession in the South and Border States, what you actually had was almost the opposite: rich planters voting for the Constitutional Union Party, the People voting secession, and conspiracies hatched by Lincoln to use Unionists to mount coups d' etat against the People in the Border States, and then later on to send a million bayonets south to war down the People under an assumed and spurious authority, to overrule and overmaster the People, who had lawfully expressed their wishes in elections, referenda, and constitutional conventions.
Your hero was a tyrant in the classical Greek sense.
Where my people came from, East Tennessee, the people were against secession. The whole state voted it down in January, 1861. But the state legislature and governor disregarded the will of the people and cut ties to the Union and invited the Confederate army into the state. Maybe I don't know about other states, but in Tennessee secession was a creation of the moneyed slave interests of Middle and West Tennessee. They later held an election to ratify their irregular action, but an election with an army in the midst is open to question. Even with Tennessee being occupied by alien forces, East Tennessee still voted to remain loyal to the Union. Abraham Lincoln had no greater friends that the loyal Union people of East Tennessee and the people of East Tennessee had no greater friend and champion than Abraham Lincoln who always sought to liberate East Tennessee from its oppressors.