Or it protects it. I remember Jim Crow. Some of my friends still have Jim Crow-era bathrooms.
If the Civil Rights Act was unconstitutional, it would be gone by now.
A law that specifically forbids the freedom not to associate is not in support of freedom of association. The claim that it would be gone by now if it were unconstitutional is laughable, given how much of the federal government activities fall into that category and remain unmolested by judicial decree.