I would of been thinking similar things, and watching the guy on the left which might have been there in case the white person reacted.
It’s almost never the guy that is instigating the situation you have to worry about, but his buddies beside or behind you.
Yes, it's often a pack attack.