Let's take a different example. On average, men are bigger than women, right? If you disagree with this assertion, then skip the rest of this reply -- we are too far off the same wavelength to have a useful discussion on this.
Ok - still there? Good. Now is Tom Daschle bigger than Oprah Winfrey because he is a man? Well, no, he's smaller. Ok - is Colin Powell bigger than Condi Rice because he is a man? Well, he's bigger. But being a man doesn't make him necessarily bigger. Is the world's largest person (probably some Guiness book of records person) a man? I don't know, but likely, because at the fringes especially, the minor statistical difference in size results in a large difference in chances that the extreme individual will be of the group that tends slightly toward that extreme. But for any given person, except a few, there are larger and smaller individuals of both genders..
Another example of the fringes of a population most clearly showing the minor difference in an overlapping attribute - what's the chances that there will ever be a woman basketball player of the size and strength of Shaq? Damn slim, I'd say.
I'm not saying, given two individuals, that being black or male or this or that causes one of them to be more or less of this or that than the other.
I'm saying that given any two populations that have some distinction that is present at birth, there will be other distinctions, most visible at the extremes.