Bottom line is that not every black in the south craved freedom. No small number actually liked the cradle to grave security which the institution of slavery offered and were totally confused as to where to go and what to do when the war abolished it.
For a modern day reference, just look at the sheer number who vote Democrat when it is clear the policy of the party of slavery locks them into crappy schools, housing and neighborhoods in return for their votes and government freebies.
Most ex-slaves remained loyal to their former owners and shared cropped to help pay the carpet bagger taxes.