This is unfortunate for very different reasons than you espouse.
It seems to me that affirmative action was defined as proactive measures to repair the damage done by generations of cultural and institutionalized discrimination against definable groups. The lexicon took on other phrases, such as "leveling the playing field" to describe the ends justifying the means of affirmative action. The debate centered on the legal and appropriate methods to correct a wrong as quickly as possible to open opportunity and demonstrate success to the next generation.
I opposed affirmative action, but find it ironic often how diversity is scoffed at by conservatives that have clearly not thought out their argument or fail to observe the culture they live in. For example:
Contrast an argument against Black Entertainment Television with the established White Entertainment Television - and then the "diversity" argument against conservative media (talk radio, FNC) and the established liberal media.
As a conservative, I am a big believer of diversity. I can't stand the gender impolitic stereotype "yes men" or Stepford group identity that often characterizes established organizations, and institutions. I am convinced that we are individualistically different, there are differences that can be identified as valuable, and individuals with valuable differences that are missing from your organization should be recruited and admitted until a critical mass that meets the organizational need is met. Being black alone plays an increasingly small role in your valuable and/or unique contribution. What's missing in many organizations, especially those that operate globally, are traits to communicate and compete in the Middle East and Asia. However, where you skin pigmentation or gender gives your company an advantage in these regions, combined with your skill in developing, producing or marketing a product/service, your diversity should definitely be considered.
Just wanted to add my perspective before the issue became too black and white.