You have a point. Maybe he isn't pro gay--maybe he's just a freaking moron. After all, "free association" is an amazingly difficult concept to grasp. It takes a real Einstein.
The right of free association only gets you so far. It does not allow you to discriminate against people in employment and housing. We've got all kinds of laws on the books that prohibit that kind of discrimination. And they've been upheld as Constitutional.
This particular case was pretty much a slam dunk.