The slavers were the bullies and aggressors in your analogy.
The North had slave states also, and did not intend to end slavery. What’s your point?