They also destroyed a lot of the built up wealth in the South.
Just freeing the slaves destroyed 5 billion dollars worth of assets in the economic reality of that era. Deaths of 20% of their male population is worth quite a lot in economic value. Destruction of their industries and the rise of foreign competition (due to artificial economic conditions created by the Union blockade) also cost them a lot of future economic activity.