Remember too that at the time women had no vote and Confederate veterans were slow to get their vote back and thus they were politically disenfranchised. Thus the rise of the underground movements like the Klan which arose in areas which had suffered great amounts of looting, rape, and murder at the hands of the occupying forces. Read about the sack of Athens, Alabama which occurred during the war and then study the Athens-Pulaski axis in the formation of the Klan. The Klan that arose as a fad after the hit film BIRTH OF A NATION had its strongest numbers in the midwestern states.
Reconstruction was a tragedy for all involved and would have been conducted much differently had Lincoln lived.