I don't disagree with the poll numbers, but, I bet if they would frame questions around how you would prevent illegal immigation and the consequences of such actions, the numbers would get much more squishy. There are those beholden to cheap labor.
Many people see this issue as only preventing terrorists from getting into the country. It is also about protecting jobs and a whole host of other things like taxes, healthcare costs, social security costs, and cultural issues.