You’re probably right about the numbers.
What incensed southerners was the fact that northerners would try to prevent them from recapturing their “property.” For some reason this really infuriated them.
In a recent book I read, southerners in northern prisons wrote home about their fears that they would be “exchanged” for a black Union soldier. They would all, quite literally, rather stay in prison or die there than be exchanged for a black man.
Racial attitude. They went into the war thinking they were worth any ten Yankees. Just imagine what they thought they were worth compared to Black Yankees.