So there were 6 or 7 “races” of humans (including the Hobbits?) and we killed off all (most?) of the ugly ones.
We interbred with these smaller isolated populations.
Smaller populations melt into the larger unless there is something that makes them want to keep separate.
like jack handy said
“I can picture in my mind a world without war, a world without hate. And I can picture us attacking that world, because they’d never expect it.”