That's because the Emacipation Proclamation had freed them the year before.
Slave ownership devolved on about 1/3 of southerners and on about 50% in SC and MS. But the vast majority of whites supported the war because they were unalterably opposed to negro equality.
The slave --holders-- were able to drive their section into treason and rebellion because of perceived threats to the expansion of slavery -- a direct threat to their financial well-being based on what the federal --government-- MIGHT do in the future. Their real enemy was demographics.
The poor whites -- what Churchill called the "mean whites", were JUST as interesting in opposing what the federal government MIGHT do, because it MIGHT devalue their most important possession -- a white skin.
Walt