I suppose you have not read about the carpet baggers.
I'm merely pointing out that their suffering was minimal when compared to the losing side in other great civil wars.
For instance, after the war was over the Spanish winners executed something over 50,000 of the losers, out of a population somewhat smaller than the USA in the 1860s. That compares with - one - such execution in the United States.
Spain had hundreds of thousands imprisoned for a decade or more after the war, as opposed to few or none in the South.
The winners in other great civil wars (English, Chinese, Russian, etc.) behaved similarly with regard to confiscation of property, execution of opponents, etc.