It is one thing to from the outside champion free speech, but quite another to knowingly use that as a rationale for hosting idiots speech. To say they have a right to speak is something different than you handing them the megaphone.
You are the same as the social media people who want to take the megaphone away from the people they don't like.
Political speech, by anyone, is the very speech the founders intended to protect with the First Amendment.
Are you afraid that you are such a weak-minded person that you would become a Nazi if you listened to one?