I can see no amount of explanation is going to make you understand but.......one last try.
Two outside groups come to your town WITH THE INTENTION of causing trouble. The SC said you don't have to allow it.
well then the SC must be racist....(lets assume the town denied a permit in this case)....becuase how are nazi's or anyone for that matter "doing harm" by peacfully protesting something..unless of course the conclusion is it will cause "trouble"
"We can't give you that permit Sir because we all know the results(the blacks.or whoever.. will riot)"