(That presumes that all the slaves were in states that left the union, and that all the business property invested in union states had nothing to do with slavery, but the disparity is so stark, the argument survives the inaccuracy of the statistics.)
A letter by a Federal officer:
Col. Giles Smith commanded the First Brigade and Col. T. Kilby Smith, Fifty-fourth Ohio, the Fourth. I communicated to these officers General Sherman's orders and charged Colonel Smith, Fifty-fourth Ohio, specially with the duty of clearing away the road to the crossing and getting it into the best condition for effecting our crossing that he possibly could. The work was vigorously pressed under his immediate supervision and orders, and he devoted himself to it with as much energy and activity as any living man could employ. It had to be prosecuted under the fire of the enemy's sharpshooters, protected as well as the men might be by our skirmishers on the bank, who were ordered to keep up so vigorous a fire that the enemy should not dare to lift their heads above their rifle-pits; but the enemy, and especially their armed negroes, did dare to rise and fire, and did serious execution upon our men. The casualties in the brigade were 11 killed, 40 wounded, and 4 missing; aggregate, 55. Very respectfully, your obedient servant,
D. STUART, Brigadier-General, Commanding."
Congressional Testimony of Nathan Bedford Forrest before the Congress Joint Select Committee, Ku Klux conspiracy: Report of the joint Select Committee to Inquire into the Condition of Affairs in the Late Insurrectionary States, Made to Both Houses of Congress. 42nd Congress; 2nd session, Senate Report 41, Volume 13; Washington, D.C. (1872) Nathan Bedford Forrest who had many Black Confederates with him Nathan Beford Forrest
I said to forty-five colored fellows on my plantation that it was a war upon slavery, and that I was going into the army; that if they would go with me, if we got whipped they would be free anyhow, and that if we succeeded and slavery was perpetuated, if they would act faithfully to me to the end of the war, I would set them free . . . Eighteen months before the war closed I was satisfied that we were going to be defeated and, and I gave them their free papers, for I fear I might be killed . . . No finer Confederates rode with me. {Emphasis Added}
When Forrest surrendered, there were 65 Free Men of Color riding with his forces. It is noted that all of these men rode with Gen. Forrest until the end of the war and not one ever signed an Oath of Loyalty or was “reconstructed.”