Exceptions established by the courts to the First Amendment protections include the following: Defamation; Causing panic; Fighting words; Incitement to crime; Sedition; Obscenity.
Fighting words: In the famous case of Chaplinsky v. New Hampshire, the U.S. Supreme Court held that the First Amendment does not protect "fighting words -- those which by their very utterance inflict injury or tend to incite an immediate breach of the peace." (315 U.S. 568, 572 [1942])
Note that the harm involved is physical harm caused by someone else who was provoked by the speaker whose speech is being suppressed. The fact that someone else flies into a rage and causes physical harm results in justifying suppression of speech by another person.
It is simply not a cut-and-dried issue.
Well, the furore over this visit by the Nazis would certainly seem to make the Nazis' appearance qualify for being suppressed under that rule. Maybe this is the point. Or maybe they just wanna raise hell.