As a melanin enriched conservative myself, my guess is he's not getting all that much support from blacks.
Any black Democrat who's ready to 'cross the line', won't stop at Carson, simply because of his skin color. That sort of voter has already abandoned racial politics altogether, and is ready to give their vote to the guy they perceive to be the odds-on favorite to actually deliver what he's promised.
At least for this election.
That makes sense.