Well, based on the picture the women were pretty ugly. I wonder what the men looked like?
Human women might have had an advantage sexually, because of better neoteny (looking younger, delicate features, thinner, more child-like face). Human men and Neandertal men were both attracted to human women, hence some interbreeding.
Human men rarely touched Neandertal women. Children of human women with Neandertal men, especially if by rape, were prone to be killed. Humans kill the male hybrids, keep the female hybrids for breeding.
This explains why there are no Neanderthal mitrochondrial or Y-chromosome lineages, along with some Neanderthal DNA. Those mixed-species girls would have bred if fertile, because even the less fit females are acceptable to some male at some time.