I went to a New England prep school with some very wealthy kids and they weren't none too bright.
The correlation is certainly not perfect. The question is do you want to be turned down for a job because your parents were wealthy. It's one thing to allow that for minorities, given the legal discrimination they suffered historically. Income-based discrimination is a dangerous new frontier.