Walter (E) Williams just wrote a column on freedom of association. His take is that you either support freedom of association totally, or that you totally throw it out. No cherry picking.
This article can be found on JewishWorldReview.com.