Well let's get back to your history for a moment. Can you list, oh, five Union generals who were slave owners? Since you said that virtually all of them were then that shouldn't be hard to do. Feel free to list more if you like.
In 1864, Cleburne, one of the better southern generals, suggested freeing and arming the slaves. The army and government went nuts, and he never was promoted further, despite the desperate need for good officers. Davis went so far as to send a personal envoy to the western army to make sure the rot hadn’t spread.
Shelby Foote, not exactly an example of a pro-Northern historian, has an interesting discussion of how CSA congressmen fought bitterly against the idea of arming blacks as soldiers - in spring 1865!
They gave in only when RE Lee appealed personally to the Congress. Only a few blacks were inducted and only a few were armed and started training before the war ended.
Many blacks indeed served in the southern armies. Most of them served in exactly the same way as the horses. As chattel used by their owners as they saw fit, not at their own volition. It is entirely possible some would have voluntarily fought for the CSA, but they weren’t given that choice, as they weren’t given any others in their lives.