And interbreeding with them! Or not, as that little scientific debate always seems go back and forth every few years.
That was settled by the DNA evidence: the data show that non-African modern humans share 1-4% of their DNA with Neanderthals, so there was indeed some hanky-panky going on.