D.Rider
Neither side treated prisoners very well, but the problem really started when the North stopped prisoner exchange when it became an advantage for them not to exchange. This caused the big buildup of prisoners, overcrowding of prisons, food supply problems, easily transmitted sickness, and death.
I got interested in the subject some years ago because my wife's great grandfather died as a prisoner at Point Lookout Prison in Maryland, the prison mentioned in the article that started this thread.
In early 1864 after the North stopped exchanging prisoners, the South offered to let Northern surgeons come through the lines with medicines to treat Federal prisoners in Southern prisons, act as commissaries to the prisoners, and report on prisoner treatment. This offer was turned down at very high levels in the Lincoln Administration and consequently thousands of prisoners died. The South offered to purchase medicines at two or three times their cost and let Northern doctors use them to treat Federal prisoners in Southern prisons (the blockade had stopped much of the flow of medicines). The North wouldn't do it.
The North basically sacrified a large number of their soldiers in Southern prisons to keep exchanged Southern prisoners from going back into the Confederate army. The North had an excess of manpower, the South didn't. One could argue that refusing to exchange prisoners shortened the war. However, Walt Whitman complained about it and rightly laid the blame for Federal prisoners dying in Southern prisons on the Lincoln Administration.
The South eventually released a large batch of prisoners in Savannah including 5,000 well prisoners without demanding prisoners in exchange.
Various arguments were put forth by the Feds as to why they wouldn't exchange troops any more. The quote I cited by Grant was one. Another was that since the South wouldn't exchange blacks, there wouldn't be any exchange. The South (Robert E. Lee, as I remember) argued back that they did exchange free black soldiers, but they would not exchange any former Southern slaves that had escaped from their masters and gone into the Federal army. They would be returned to their Southern masters. After all, it was still US law that escaped slaves must be returned to their owners.
The black prisoner exchange argument was specious. Federal General "Beast" Butler, Commissioner of Exchange for the North, admitted being prepared to keep preventing exchange on one pretext or another even if the South agreed to all demands of the North. They didn't want the South to get the extra manpower.