Only us white folk....
At least I have an excuse now... :)
It looks like, based upon the numbers, that interbreeding wasn't all that common - and based upon genetic markers - most likely almost exclusively male Homo sapiens and female Homo neanderthal.
So “crowded out” also means, most likely, military domination and the theft of any surviving women - some of whom were ‘recruited’ into child production.