The whole USCT affair was the excuse given. The federals calculated (correctly) that since they had more men to begin with, ending the parole system that had previously been in place would affect the South more. They could much less afford to lose men. The end result was pretty brutal for POWs on both sides but it was an effective tactic.
its true that not much by way of foodstuffs had to be imported by the North. There were some medical supplies that came in. The South, being an economy based on producing cash crops for exports rather than subsistence farming, had a harder time switching over to completely different crops and found it especially difficult to bring in medical supplies. Given that critical shortages of food and medicine, its no wonder that federal POWs fared badly. Note that that was not the result of a policy of deliberate malice. For Confederate POW's in the North however...there were no food or critical medical supply shortages in the North. The astronomically high death toll at what the federals' own reports accurately called "death camps" like Camp Douglas, Elmira (aka Hellmira) or Point Lookout were the result of deliberate malice.
“The whole USCT affair was the excuse given.”
It was the reason. In May 1863 the Confederate Congress passed a resolution, endorsed by Davis to turn captured USCT’s over to the states as escaped slaves, rather than treat the as POWs. White officers were to be tried in state courts for inciting slave insurrection. The Lincoln administrations position was that USCT were soldiers and entitled to be treated as any other prisoner of war. Breaking off the prisoner exchange discussions was politically unpopular in the North as a large % of Northerners support prisoner exchange. Informal discussions on the issue continued without breaking the deadlock. Several prisoner exchanges were conducted, primarily wounded prisoners for wounded prisoners. At no time would the Confederates exchange USCTs for any reason. When Grant assumed command of all the Union Armies in 1864, he ended any hope of restarting the prisoner exchange program. Pretty much for the reasons you gave as the reason for the original end of prisoner exchange. In spite of the Confederate announcement their plans for USCT prisoners, they actually did not carry out their stated actions. USCTs were retained in their prison camps with other Union prisoners. But they were never exchanged at any time for the remainder of the war.