Is saying “Oh, just shut the f*** up!” Racist?
Yes. But if you don't care, then it doesn't matter.
'Tis why the left hates Pres. Trump so much. They can yell "Racist! (or bigot, sexist, etc) at him until they're blue in the face. His response - correct response - is "I Don't Care."
Takes all the power away from them. All they can do is yell louder. Good negotiating technique, and one that any parent with a 3-year-old is familiar with.