Then you aren't very perceptive. The rabid hysterics are not the only people who "complained about and warn against" illegal immigration. They are, however, the people who made the task of finding a reasonable solution impossible. Nothing less than the most Draconian solutions will satisfy them, and their snarling pack behavior in turning even on their own political allies alienated wide swaths of the American public.
But, of course, in the world you and your ilk occupy, you are never responsible for the havoc you wreak. Some other guy always is responsible for all your sour grievances.