I don't buy that. Most Southerners supported secession. They voted for it in several states and elected delegates to do so in others. Those who fought for the union were in many cases disowned by their families.
100,000 Sputherners fought for the Union. A majority in West Virginia, and the Apalachians. Also strong in Notlrtjern Georgia, and Alabama. And more would have been Unionists under Jackson.