Actually, there is a lot more to the survey done. It looked at 5 cities, Phoenix, Las Vegas, Chicago, DC, Raleigh-Durham. The survey showed that Phoenix and Las Vegas, the two with the highest concentrations of Latinos considered it a problem, with Phoenix calling immigration a very serious problem. So that does seem to negate somewhat the bias error attributed to the zip codes that you pointed out.
Your hypothesis is very possible, imho. A zip code has about 10,000 residents, and they certainly can be ghettos.