Isn't that, like, an act of war?
“like” NO.
Their being there were acts of aggression against the states. They were there by the grace of the sovereign states, and when that grace was no longer granted, the feds had to make a decision: would their right to remain, even where not wanted, overrule the right of the states to protect their domains?
In choosing to place the rights of the federal government over the rights of the states, the “Union” became less a voluntary republic than a centralized tyranny. The conflict was as old as the country itself.