Sadly, you’re right.
I think that “White Supremacist” should be fighting words as well. Along with insults to family members such as one’s mom if said by an actual enemy, and any words or actions against ones’ minor children.
But especially White Supremacist because that is an impugnment of honor that can have concrete impacts on one’s reputation and ability to earn a living. So to be falsely branded such should be worth at least a stiff poke in the nose.