Wrong, the discussions on prisoner exchange were broken off because the Confederate representatives refused to accept black Union soldiers as legitimate prisoners of war.
That was a pretext. The union army high command (correctly) calculated that ending the prisoner exchange would benefit them due to their much larger population.
Who in the Union Army High Command?