Now, that's different. Either disband them all (including the Congressional Black Caucus), or allow them all. The 'freedom of association' thing is a totally separate issue from the 'freedom of speech' thing. (The quotes being there because we're not talking about their legality.)
Again, when my fellow Cardinals fans get together, we will say with impunity that the Cards' pitching s*cks. But, we won't let a steenkin Cubs fan say it about our Cardinals without a fight. ;O)