Brown sugar? What's racist about that?
It could be a reference to the song "Brown Sugar" which tells of a slave owner who takes sexual satisfaction from his slave girl. Or it could be a reference to the product, brown sugar, which is actually white sugar mixed with molasses (in other words, white on the inside but made to appear brown.) Get the idea?