Oh I had that same conversation with members of the Black Engineers Society when I was at Michigan State. Most agreed with the rationale that minority groups these days do more to polarize race relations than help. But, someone else said the reason that only white people can be racists is because white people are the ones with power. I don’t recall my reaction to that, but I know it was at the very least an eye roll.
Whites look at corrupt white politicians and corporate executives and see a criminal class out to loot public or shareholder funds and live above the law.
Blacks look at corrupt white politicians and corporate executives and see whites.
(Much as many whites look at Jesse Jackson and Al Sharpton and see blacks.)
Blacks typically don't understand that the elites do not derive their power from the consent of the governed average white American. Thus the old 60's saw about "Give Up The Power, Whitey!" is nonsense - "Whitey" doesn't have it to give them. They would be better off yelling "Give up the Power, Goldman Sachs!" ;)
My answer to that silly argument:
We all have power. If you hate, then you choose to feed a power that eats away at your soul. And the souls around you that catch the disease.
If you use your power with love for yourself and others, it is very hard for people to set limits on you.
We all have the choice.