It's pretty ironic to make that argument in the defense of slaveowners.
What is really Ironic is the use force to subjugate people and claim you did it to stop subjugation.
Especially when your side was okay with subjugation when it was funneling money through the right hands in New York and Washington DC.