Yeah...that's not racist at all.
Actually, that is NOT racist.
Now, if he had said, "We talk to each other differently than we talk to YOU PEOPLE", well, that would have been an obvious racist slur.
/s
Actually:
Willie Clark, president of the N.A.A.C.P. branch in San Bernadino, Calif., said the overall tone of Mr. Perot's remarks and particularly his use of the phrase "your people" reflected how culturally out of touch he was with his audience.
"When he said 'you people' or 'your people,' it was like waving a red flag in front of a bull," he said. "It's something white folks have used when they don't want to call you nigger, but they don't want to treat you like an equal."
-- New York Times: July 12, 1992
Archie Bunker to George Jefferson: “You People?! Who are use callin’ You People?....You People is You People!!”