In your personal life. I'm speaking only of businesses that are public accommodation. You can choose not to associate with blacks, women, or Hindus but you don't have a right to refuse service to them unless you are a private club.
We are all equal under the law. Renting to a gay person does not make that person more equal.
So you can associate with whoever you want only in your personal life. Now, since we live in homes and work in offices if you must rent or sell to anyone regardless of who they are, and you must employ anyone regardless of who they are, where are you truly able to exercise freedom of association?
It all wraps together. There is more threat to freedom of association than you are recognizing and less freedom than you think. Basically, whenever it matters, politically, economically, or personally, other than in your own home, you can’t choose who to associate with.