Well I expect that George Herbert Walker Bush would probably get his butt kicked in East Los Angelos if he referred to his grand children as "the little brown ones". I probably wouldn't though.
I think if went up to someone in any part of Los Angeles and said to a Hispanic person and said"Hello, little brown person" you would end up in the hospital. Its a racist statement pure and simple. Its like saying the "n" word. Calling a Hispanic "brown people" is racist and their "n" word. What Karl Rove said was racist.