"A woman with fair opportunities, and without an absolute hump, may marry whom she likes." -- William Makepeace Thackeray
IOW ugliness doesn't stop a woman from reproducing.
I don't understand why this is so hard to get.
You need more than mere reproduction. You need the reproduction to be in the human community, because if it's over in the Neanderthal community it's of no consequence to the issue at hand.
And then, you need for the infant to grow to adulthood. In other words, you need to the infant to be raised by humans.
And, presumably, it will help to be able to talk to one another.
Otherwise, you're left with the occasional rape. And that's of course making the strained assumption that the Neanderthals even wanted to have sex with us.
We were probably just as butt ugly to the Neanderthals as they look to us.