As long as their agenda doesn’t involve forcing anything on anybody, I can think of no reason to ban them. What they do in their own homes is none of our business.
If I were to start a group called "We love intercourse in the missionary position! (WLIMP)" there would be a much bigger pool of potential voters. But they'd also rightly think I was a goofball for starting such a group.