Have you listened to him very much? Maybe you heard him at the particular times he seemed to do that? I haven't noticed that particularly, he definitely makes jokes that are not politically correct; OTOH I have heard him give a lot of credit and respect for black people who are not shackled by anger and the victim mentality, and therefore can take advantage of opportunities that America has to offer.
I am not black, but lived in Hawaii for around 14 years where "haoles" are in the minority - especially on the Big Island. And I did experience people giving me "stink eye" just because I was white, being rude, etc. And it is not fun to experience. I also learned that people who are racially prejudiced carry around anger and hate in their hearts, and those who are the targets don't have to. Nothing poisons a person - even their physical health - like hate and anger.