The North didn't want to exchange prisoners because to do so would help the Southern manpower situation. The North had stopped prisoner exchange long before the Lee-Grant communications.
Here is an admission of the Federal Commissioner of Exchange, General Benjamin "Beast" Butler, concerning prisoner exchange:
In case the Confederate authorities should yield to the argument...and formally notify me that their slaves captured in our uniform would be exchanged as other soldiers were, and that they were ready to return to us all our prisoners at Andersonville and elsewhere in exchange for theirs, I had determined, with the consent of the lieutenant-general [Grant], as a last resort, in order to prevent exchange, to demand that the outlawry against me should be formally reversed and apologized for before I would further negotiate the exchange of prisoners.It may be remarked here that the rebels were willing enough to exchange prisoners at this time, man for man, were we to permit it to be done.
The North would not permit the South to allow Northern doctors through the lines to bring medicines and treat Northern prisoners in Confederate prisons even though the South offered to buy the medicines for the Northern prisoners. Sad, but true.
As I stated in post #586: "Northern politicians played politics with Andersonville then just as your northern politicians play politics with Abu Gurab and Gittmo now."