Once you've worked with the general public you realize that while humans produce the occasional Einstein, nature produces a lot of examples at the other end of the bell curve too. I would be willing to bet "you people" had nothing to do with race, and everything to do with the guy self-identifying with the idiot end of the bell curve.
Unfortunately for the manager, he didn't mumble his "you people" retort in Spanish.