I don't know about physically being hurt...I was more concerned about the "in your face" shouting. I did have a bad time getting past the Supreme Court Building last inaugural wearing a red satin cowboy hat etc.
Those groups get stirred up and need a target to "spew-upon".
I don't want to be available to them...like all bullies they pick on the weakness of not having numbers for support.
Frankly I think the best thing to do if one of these folks starts ragging on you is just to laugh in their face. Derision is the best legal weapon we have.