I stand by my comments.
The party declined rapidly in the North in 1855–56. In the Election of 1856, it was bitterly divided over slavery. One faction supported the ticket of presidential nominee Millard Fillmore and vice-presidential nominee Andrew Jackson Donelson, who won 23% of the popular vote and Maryland's 8 electoral votes. Fillmore did not win enough votes in Pennsylvania to block Democrat James Buchanan from the White House. Most of the anti-slavery members of the American Party joined the Republican Party after the controversial Dred Scott ruling occurred. The pro-slavery wing of the American Party remained strong on the local and state levels in a few southern states, but by the Election of 1860, they were no longer a serious national political movement..Some historians argue that in the South the Know Nothings were fundamentally different from their northern counterparts, and were motivated less by nativism or anti-Catholicism than by conservative Unionism (preserving the Union of states rather than labor unions); southern Know Nothings were mostly old Whigs who were worried about both the pro-slavery extremism of the Democrats and the emergence of the anti-slavery Republican party in the North
Note also these comments:
The term "Know Nothing" is better remembered than the party itself. In the late 19th century Democrats would damn the Republicans as "Know Nothings" in order to secure the votes of Catholics. Since the early 20th century, the term has been a provocative slur, suggesting the opponent is both nativist and ignorant. In 2006, an editorial in the neoconservative magazine The Weekly Standard by William Kristol attacked populist Republicans for not recognizing the danger of "turning the GOP into an anti-immigration, Know-Nothing party."[4]Citizen Know Nothing, image of the Know Nothing party's nativist ideal
The lead editorial of the New York Times for Sunday, May 20, 2007, on a proposed immigration bill, referred to "this generation's Know-Nothings...."
Bolding added for emphasis. Slinging around the phrase "Know Nothing" indiscriminately is aiding and abetting the Democratic, er, Socialist-Communist, Party. And adding the accusation of racism is ludicrous: So were all the other political parties except for a small fraction of the Republicans later called the Radicals. The Know-Nothings were probably less "racist" than the Northern War Democrats.
It makes no sense to distort political history from the viewpoint of 1960's Civil Rights radicalism, and use a misunderstanding of the phrase to twist the distortion even further.
And it's funny to listen to Kristol's rap, because the GOP started out being mostly anti-immigration, as illegal immigration barely existed then, and the overwhelmingly Protestant Republicans viewed the Catholics within a broad spectrum of distrust, with the Know-Nothings being just one end of that spectrum.