The only way you can limit access is to make it a private (ie: members only) club.
First of let me say that a number of private clubs have been forcibly integrated by the courts. Gender and not race has been the main cause for such actions. But more importantly here, is the idea that private clubs is the "only way." Imagine if some one had said that due to the Jim Crow laws, the only way to have integrated access to a restaurant, swimming pool, or movie theater was to make private (i.e. members only) clubs. Obviously you are not correct on this one, as we now have integrated them with out resorting to such clubs as the means.
So also the opposite could occur.
The "private clubs" that were forcibly integrated were done on the grounds that they were "private" in name only. There still are a number of "no blacks" private golf courses in the US. If you claim the "private club" exemption, you really need to closer to "private" than "public".
IIRC, SCOTUS ruled the Civil Rights Act of 1964 to be Constitutional. If that's the case, then it doesn't matter what anyone else thinks.