Lincoln’s unwillingness to compromise on principles likely led to more deaths of prisoners of war, on both sides; after he quit agreeing to anymore exchanges of prisoners as long as the Confederate government continued to insist on separate and unequal terms for black prisoners.
During that period conditions at the notorious Andersonville prison camp in Georgia continued to deteriorate - while northern ministers pleaded with Lincoln to agree with the south’s terms on prisoner exchanges - and it was during that time that an uncle of my mom’s grandfather died - from starvation followed by suicide - at Andersonville.
Conditions at prisoner of war camps were horrible, on both sides of the war. Disease and illness was rampant in the northern camps and food was scarce in the south. Andersonville just happened to be one of the worst, in part because of its very large size. And as much as the suffering moved Lincoln personally, he would not agree to separate terms for black prisoners. He argued that doing so made a mockery of the war effort itself.
It was as unpopular of a decision in the north as it was in the south. I think Lincoln carried within himself his own suffering for some decisions he felt he had to make, unpopular as they sometimes were, and knowing they did not relieve suffering he knew would continue. It is the suffering Lincoln that I see in the late photos of him.
You've mistaken a "peg", a "McGuffin", for a real reason.
Very wilful people intent on getting their way always throw out a plausible reason -- a peg -- for their obduracy, to fob off objectors and reasonable people who recognize unreasonableness when they see it. That's what Lincoln did with everything, and FDR and Lyndon Johnson after him.
Lincoln refused to continue prisoner exchanges because he knew that the net military advantage would accrue to the North despite the grisly experiences awaiting prisoners as they piled up in the camps. He was determined to hang on to every Southern prisoner and prevent Southerners' return to service. That was the only reason for his decision, and its consequences.