Curiously, you failed to mention the most important fact: that the "Know-Nothings" were anti-Catholic.
Anti-immigrant, anti-Catholic, same difference in the 19th century politics of the know nothings. The point being they had an animus towards immigrants.