More ex-slave refugees died of disease, exposure and starvation (@ 400,000 total) than soldiers died on either side from any cause (250,000 CSA, 350,000 Union). Union neglect of ex-slaves fleeing behind their lines was at least as lethal as Civil War combat to both sides combined.
Sick From Freedom explains this in detail. Or read the review at the Journal of Military History.
Better, buy the book from Amazon - the Kindle edition is only $12.09, and you can read it on your computer with the free Kindle application.
Do math much?
Read the review in The Michigan War Studies Review. The reviewer comes up with a lot of criticisms.
Apparently Jim Downs, the author, has a very modern welfarist idea of the freed slaves deserving food, shelter, and medical care from the government. Whether that idea is right or not, it would have been impossible for 19th century America to provide that level of care.
What is surprising is the degree to which food, housing, medical care, and employment were provided by the Army and Freedmen's Bureau in an age unused to such mass provision of care. "Humanitarian disasters" have been common and familiar in our own day. It's not surprising that an earlier era didn't know how to cope with large numbers of refugess.
While there were doubtless some terrible cases of negligence or callous indifference, I doubt what happened could truly be called genocide. I don't think you can blame either the US Army or the federal government for smallpox epidemics that killed off thousands -- including many people who weren't freed slaves or African-Americans. Maybe we should blame the former slave owners for not having inoculated their work force when they were in charge of things.
Someone could do more with the topic, though. I don't know how many African-Americans died in the epidemics of the 1860s, but the losses and their effect on the country are probably something historians would be interested in. Epidemics in the western territories, Western Canada, the Pacific Islands, China, and India during roughly the same years did a lot to shape the history of those countries or regions.