Interestingly, my liberal constitutional law professor suggested a better constitutional grounding might have been the post-Civil War civil rights amendments. They chose not to go with that theory, however, because there was no case law and they felt they had a better chance under the Commerce Clause.
The Court has been in abeyance since FDR’s era.