Point of reference is an invalid argument. That makes everybody right all the time.
The only valid argument is that man has a God given right to be free. It is the basic premise of the Declaration of Independence and the foundation of a Constitution written specifically to limit the powers of government.
When oppressed, one has the right, even the obligation, to seek a remedy, regardless of what a man-made law states.
So black slaves had a right and obligation to rebel, and those who would have opposed them were in the wrong?
A word of caution. It isn't possible to make a case that slaveowners fought in defense of liberty. Unless you wish to assert that their slaves weren't really men, as the US Supreme Court ruled was the law of the land.