When HS spread into Europe and probably also with Asia, the population of Neanderthals in Europe and Denisovans in Asia were very sparse. The incoming Moderns were many times their number so the genes contributed by the older groups to Man were a correspondingly tiny percentage.
That’s just an out-of-Africa myth.