Where there even 50,000 total confederate prisoners held by the Union during the entire war? Anybody seen any figures on this? Most mass surrenders, like Donalson or Vicksburg, led to quick exchanges.
No aspect of the American Civil War left behind a greater legacy of bitterness and acrimony than the treatment of prisoners of war. "Andersonville" still conjures up images of horror unmatched in American History. And although Northern partisans still invoke the infamous Southern camp to defame the Confederacy, the Union had its share of equally horrific camps. Prison camps on both sides produced scenes of wretched, disease-ridden and emaciated prisoners as repulsive as any to come out of the Second World War.
Partisans in both the North and the South produced wildly exaggerated novels, reminiscences of prisoners, journalistic accounts and even official government reports which charged the enemy with wanton criminal policies of murderous intent. It took several decades for Revisionist historians to separate fact from propagandistic fancy and deliberate distortion from misunderstanding. Even today the bitter legacy of hate lingers on in widespread but often grossly distorted accounts from this tragic chapter of American history.
Neither side deliberately set out to maltreat prisoners. Arrangements were made hurriedly to deal with unexpected masses of men. As neither side expected the war to last long, these measures were only makeshifts undertaken with minimum expenditure. Management was bad on both sides, but worse in the South owing to poorer, more decentralized organization and more meager resources. Thus, prisoners held by the Union were somewhat better off.
In the first phase of the war, 1861-1862, the relatively small numbers of prisoners taken by both sides were well treated. Both sides agreed to a prisoner exchange arrangement which operated during the latter half of 1862. Under the cartel, captives remaining after the exchanges were paroled. But the agreement broke down, in part because of Northern refusal to recognize the Confederate authorities as anything other than "rebels," and in part over the Negro question.
''In a war of this kind, words are things. If we must address Davis as president of the Confederacy, we cannot exchange and the prisoners should not wish it," declared the influential Harper's Weekly.
Following the promulgation of the Emancipation Proclamation on New Year's Day, 1863, the North began enlisting former slaves into the Federal army. Confederate President Jefferson Davis declared that "all Negro slaves captured in arms" and their White officers should be delivered over to the South to be dealt with according to law. That could mean rigorous prosecution under strict laws relating to Negro insurrections.
Still, special exchanges on a reduced scale continued, but from 1863 onwards, both sides were holding large numbers of prisoners.
On 17 April 1864, General Grant ordered that no more Confederate prisoners were to be paroled or exchanged until there were released a sufficient number of Union officers and men to equal the parolees at Vicksburg and Port Hudson and unless the Confederate authorities would agree to make no distinction whatsoever between White and Negro prisoners.
On 10 August, the Confederate government offered to exchange officer for officer and man for man, accompanying the proposal with a statement on conditions at Andersonville. This offer induced General Grant to reveal his real reason for refusing any further exchanges. "Every man we hold, when released on parole or otherwise," Grant reported to Washington, "becomes an active soldier against us at once either directly or indirectly. If we commence a system of exchange which liberates all prisoners taken, we will have to fight on until the whole South is exterminated. If we hold those caught they amount to no more than dead men. At this particular time to release all rebel prisoners North would insure Sherman's defeat and would compromise our safety here." (Rhodes, pp499-500)
In October, Lee proposed to Grant another man-to-man exchange of prisoners. Grant asked whether Lee would turn over Negro troops "the same as White soldiers?" When Lee declared that "Negroes belonging to our citizens are not considered subjects of exchange," the negotiations completely broke down.
http://www.ihr.org/jhr/v02/v02p137_Weber.html
The article is very balanced regarding POW treatment by both sides.