Nobody wanted to be drafted to fight for the “black man”. It was ok to volunteer for Lincoln’s War but being drafted was a whole different kettle of fish.
Oddly, the insurrection resorted to the drafted long before the US.
And noone really wanted to fight for slavery either, as you tell us.
“A rich man’s war, but a poor man’s fight.” was the way they put it.
Slave owners got deferments. Non-slave owners were conscripted into the militia, and the militia was forced to patrol for runaway slaves long before the war started.