You are completely correct. I know many, many LEGAL Mexican residents who barely speak English. I disapprove, I think (from my personal experience) that if you plan on spending any time in another country, you should learn to speak the language.
Unfortunately, many of the LEGAL Hispanic residents don't speak English.
I think we look xenophobic if we simply make language the litmus test of who should be here and who shouldn't. You CANNOT tell who is illegal simply by looking at them.