It is the people who are attracted to those symbols that are disturbing.
When I was a kid going to gun shows there were tables and tables of Nazi memorabilia everywhere. That’s all it was, viz. memorabilia. Don’t see it much anymore at gun shows.
True, which is why such people may merit scrutiny, less so the inanimate symbols. Toward that end, there is something to be said for possible kooks displaying their affiliations. Seriously dangerous people of course tend to hide what they are about.