Nearly every Confederate state had counties or whole regions where slavery was rare to non-existent -- notably western Virginia, eastern Tennessee, western North Carolina, northern Arkansas and others, all of which remained loyal Unionists, and sent their young sons to serve the Union Army, not the Confederacy.
Those counties and regions did not consider the Union Army an "invasion force", but rather liberators who came to free both the slaves and themselves from the tyranny of Slavocratic Confederate government.
All told, Southern slave-states provided over 350,000 troops for the Union Army, at least two-thirds of whom were white.
So, why did over a million other Southerners serve in the Confederate army?
Because, like the Confederacy itself, they were willing to fight, kill or die to preserve, protect and defend their "peculiar institution", slavery.
How many soldiers from Maryland, Delaware, Kentucky and Missouri serve in the PACS? Hundreds of thousands.
Your side invaded, killed, burned, murdered and destroyed, all the while you slapped the chains of the FedGov on people who didn't want to be ruled by it.
Then after the fact, you said you did it all in a "good" cause, a cause you certainly didn't care about prior to invading, but one that suddenly became important after two years of fighting.
It was a very expensive fig leaf.