True, which is why such people may merit scrutiny, less so the inanimate symbols. Toward that end, there is something to be said for possible kooks displaying their affiliations. Seriously dangerous people of course tend to hide what they are about.
My impression about the German reenactor groups (back when I followed reenacting) was that they realized they were under intense scrutiny, took what they were doing as living historians very seriously, tended to focus more on equipment and tactics than the period politics, and kept the unhinged and problem cases out of their groups.
The most problematic reenactor group I ever witnessed was a Free Polish group.