Well, assuming the concept of freedom of association extends to freedom not to associate, both the federal and state governments, fully enforced by federal and state courts, have been denying people the right to refuse to associate on the basis of race for decades.
Of course, in some areas they are more strict, especially in public accommodation, and some areas not all, like marriage or renting rooms in your house.
My problem is I don't see a difference in the constitutional basis for permitting or prohibiting a state from overriding the right to refuse to associate because of race as distinguished from homosexuality.
homosexuality is a behavior
does a bank have a right to refuse entry to a known bank robber?
does a casino have a right to refuse entry to a known card counter?
Oh yes they do!