Did you mean to say:
"More 'black' Americans have a white slave ancestor than do 'white' Americans"?
If you have ancestors outside of New England who were here in the 16th century, or most of the 17th century, I guarantee you have some rather swarthy (black, actually) ancestors!
There is no shame in this.