Your context is irrelevant to the point that they had a right to do something, even if it was for a bad cause.
You have done exactly the same thing as saying you are against "freedom of speech" if it is speech with which you disagree, and against "freedom of religion" if it is a religion with which you disagree.
You are arguing that someone's right should be contingent upon your moral approval of their goals or purpose, and I, and the natural law philosophers, argue that rights are inherent, and they are not suspended because some people don't like the reason why other people want to exercise them.
I wasn't arguing anything, merely noting that when people say the war wasn't about slavery that's not so. Slavery was the root of the issue. If there was no slavery there would have been no wedge between north and south other than regional differences. But Slavery was the poison pill that made it inevitable that the USA would fail or would be reborn through fire and blood.
But yeah, I do feel that your right to do whatever you want has limits. You can speak about anything you want whether I agree with it or not. But you cannot enslave someone else. If you think there are circumstances where saying "you can't own slaves" is wrong because nobody has the right to put limits on your freedom and you don't see any contradiction in that position, well, we'll have to agree to disagree on that.