One doesn't have freedom of association without the power to exclude. If I choose to live among a community of those who want straight relationships, your plan precludes that option. Thus, you have canceled my right of free association.
By your definition, we already don't have free association, so I guess I'm not too sure what you're worried about. That ship has sailed.