So would you with “your guys”.
"My guys" would probably include radical abolitionists and whatever believers in racial equality there were at the time.
They didn't get along with the rest of the guys, but both bunches of guys were on the same side against your guys.