Neither army wasted competent officers or disciplined infantry units in guarding prisoners. Neither side was willing to expend resources to take care of prisoners.
Had the South been willing to accept captured black Union soldiers as legitimate prisoners of war, the exchange system that had been in place early in the war would have continue after July 1863. Places like Camp Douglas and Andersonville would not have been needed.
The North ended the parole and exchange system because they calculated that due to their huge manpower advantage, any Southern soldier taken out of circulation was well worth leaving one of their own in captivity.
It was a brutal calculation but one that is rational. Anything else was just an excuse the Union Army leadership needed to give their own civilian population for leaving their sons and brothers to starve in POW camps.
Wrong, the discussions on prisoner exchange were broken off because the Confederate representatives refused to accept black Union soldiers as legitimate prisoners of war.