In short, for all we know Neanderthals may've been inclined to kill modern humans on sight, or vice versa. There's certainly plenty of fossil evidence to establish that Neanderthal fossils vanish from any given region almost immediately after the arrival of Cro-Magnons.
We know that over and over again that tends to be the initial reaction of aboriginal humans, for that matter. It isn't 'Oh, come screw us'; it's 'Get the $%#& off our land!" Eventually, by conquest or by diplomacy, a rapport gets established, and then the screwing comes along, but to reiterate yet again, there is no reason to think that we could communicate with Neanderthal in anything but the most rudimentary fashion.
Would you want to go to sleep beside some incomprehensible wild-eyed burly ape-woman scheming how to bash your head in with a rock for all you know?
Oh, which brings me to the other point that I forgot to re-mention: In order to have Neanderthal intermixed with modern human populations, then what you have to have is not human men sleeping with Neanderthal women, but rather human women sleeping with Neanderthal men. That's because you have to have the progeny end up back in the human community. So, what you need is for human women to somehow go sleep with Neanderthal men, return or remain in the human encampment, then bring the pregnancy to term, then to raise the hybrid up to adulthood, and then to have that hybrid cross-breed back into the human community again.
Here we all are, discussing a very intruiging subject.
And you want to drag up my wild youth!
She seemed so nice at the bar...