This argument that "this doesn't affect port security in the slightest" is not concrete.
So it doesn't affect those who provide the security, however ponder this example...if a terrorist with the intent of destruction moves to Newark, it doesn't affect how the Newark Police operate, but it does increase the chance of destruction in Newark. The police still have to catch the terrorist. The probability of destruction increases.
Immigration to Newark has nothing in common with port security.