Have you heard of the “Bradley effect”? This refers to just the behavior you’re describing, where a person will say he will vote for a Black candidate, but doesn’t actually do so, leading to Black candidates polling better than they actually perform. Wiki has an interesting article on this.
According to the Wiki article, this effect is more pronounced in states where the Black population is smaller.
I hope it has to do more with the fact that he is a very scary candidate more than his race.