Perhaps the true solution is to shrug your shoulders and not care if you’re called a racist. If more people did that the “race card” would disappear. Frauds like Zero, Sharpton, Abrams, Gillum, Jackson (Is he even still alive? He has been very quite!), etc. would have no leverage over you. One could even practice MLKs dream of using individual character as your guide.
If it did not have the very real potential of damaging a man’s reputation to the detriment of future employment I would agree. But people should not get to bear false witness with impunity.