In this equation it quite likely doesn't matter what human guys will sleep with; it matters what human girls will sleep with. That's the conceptual hurdle that a lot of people seem to have difficulty vaulting over. In order for the progeny to end up in human communities, then what you almost certainly have to have human women sleep with Neanderthal men. If what you have is human men ravishing Neanderthal women, the progeny are gonna end up back over in the Neanderthal community, and needless to mention all those went dead.
Then, assuming you have human women sleep with Neanderthal men (and willing to raise the hybrid progeny) what you also need is Neanderthal men willing to sleep with human women, and we can hardly assume that would be the case. If you want an analogy, take the common chimp and the bonobo chimp: the two closely related subspecies nonetheless have radically different sexual behaviors.
Bonobos rock...How can anyone knock an existence that consists basically of one life-long orgy.
There is a long history of men taking women prisoner as sex slaves. The only question would be if it was possible to make Neanderthal women submit to such over the long term