It was considered more important to derail racial prejudice than to respect the right of freedom of association (or non-association). Now we’re where we are today, with people forced to associate or not, do business or not, based on whatever the government tells them they should do today.
The SCOTUS already ruled that Freedom of Association is precluded by the freedom NOT to associate.