The worst race riots in the history of the United States were called draft riots and took place in NYC just after the battle of Gettysburg. Any black folks unlucky enough to find themselves at that time in the streets were beaten or lynched.
It took a substantial portion of the victorious Union Army to be diverted to NYC to finally quell those riots.
A lot of people were uninterested in being conscripted to fight and die for the sake of “fighting slavery”.