"My guys" would probably include radical abolitionists and whatever believers in racial equality there were at the time.
They didn't get along with the rest of the guys, but both bunches of guys were on the same side against your guys.
Our guys, believing in liberty and justice for all, endured the atrocities until it all ended. Along with it died the concept of a nation governed by men enlightened with the concept of liberty, not egalitarianism.