"This issue wasn't even on the radar in the 2004 elections. "
Not true. Both sides staked out public positions on immigration during the Presidential race.
For example, see here:
http://www.migrationinformation.org/Feature/display.cfm?ID=202
http://www.migrationinformation.org/feature/display.cfm?ID=232
Which the public proceeded to ignore.