Four of my GG-grandfathers fought for the Confederacy- none owned slaves (only about 5% of the southern population owned slaves). Virginia, my native state, voted against secession on April 4, 1861 and did not vote for secession until Lincoln called for troops to invade the South.
And then you lost about of a quarter of your white population when the Western counties with few if any slaves told the tidewater Virginia slavocrats to go to hell.
Almost heaven, West Virginia. They were never part of the slave empire and those Hillbillys never much cared for the phony aristrocrats like Bobby Lee in the first place. ;~))