Border security is about so much more than illegal immigration. Mexico is sliding into chaos and could rapidly devolve into anything from civil war to the next Hugo Chavezdom. Not building a fence now is going to look like a apocalyptic mistake in five years time.
The more I think about it we actually need about a 10 mile DMZ zone with active armed army units engaged and ready to take anything out crossing that zone.