Keep pounding until they drop it, and then pound some more to make sure they don't do it again. Bring it up later. Be obnoxious about it. "Oh, yeah, you're the one who likes to make false accusations of racism. Made any lately?"
The reason you want to be persistent is that this is largely a hit-and-run tactic that doesn't stand up to scrutiny. Make them pay for using it. If they start to avoid you, so much the better.
Good post - I can relate. I’m all for keeping the pressure on - to make them accountable for what was said. Hammer the damn thing home until they are sick of hearing about it - and sorry they brought it up. Either put up or shut up!