Tell that to business (or churches) that have race-segregated restrooms.
Race-segregated restrooms are prohibited by the Civil Rights Act of 1964, not by the Constitution, which is why they were legal before 1964, despite the Supreme Court's decision ten years earlier that segregation by the Government was unconstitutional.