I know it is hard for many Americans to realize how different the US was in 1860, and apply today's issues to what was going on back then.
Your comment implies massive taxation by the Union. FYI, the total Federal budget in 1860 was $60,000,000. I believe this works out at about $2 per capita nationally. Even at the value of a dollar then, that won't finance much oppression. You could probably pay your annual federal taxes by working 2 days at the average salary for unskilled labor.
Even this minimal amount of taxation was entirely indirect, mostly as tariffs on imports. Don't want to pay taxes? Don't buy imported products. Not an option we have available today.
Almost all of that money went to fund the (very small) armed forces and the Post Office. US residents at the time were among the lowest-taxed people in history. Since average state taxes were lower in the South, southerners paid even less tax.
BTW, not even all the $60,000,000 came from taxes. A major source of income for the budget was the sale of public land.
Federalis freedom often meant starvation, rape by federal soldiers, looting, forced separation from their white friends who suffered the same fate.
All of this was a result of the war. A war started by the South.
Such laws were invalidated by the 14th Amendment. And a great many ex-slaves obviously did move North or West after the war. Any difficulties they faced on arrival were due to public sentiment, not legal restrictions.