What truly irritates middle class black Americans is that they are not distinguished by other Americans from poor and lower class blacks.
After years of education and hard work, being lumped together with ignorant, poor, slothful and often criminal lower classes would annoy most anybody.
But this meets head on with the egalitarianism of especially whites, who reward the achievements of education and success by leaving successful people alone. Not just educated and successful blacks, but everyone who is educated and successful. There is no “community” of the middle classes, if anything, there is isolation.
So while uneducated and poor blacks are ignored, because others don’t want to associate with them; middle class blacks are ignored out of unspoken respect. But there is no way to tell the two things apart. They assume they are ignored because they are seen as “the same” as people far below their station. And this bugs the heck out of them.
I would assume blacks get irritated by being lumped in with gays as similar minorities, also.