It need not have been so brutal. When white Americans met with American Indians, for example, peaceful interbreeding frequently occured; it wasn't all wars like in the cowboy movies.
On the other hand, Valerius Geist argues persuasively that very little Neanderthal/modern interbreeding could have taken place: Neanderthal Paradigm Very little, doesn't mean zero however; it would only take once for the necessary genes to cross over into a modern population, where they could be selectively acted on.
In small genetic groups inbreeding is not good, that is why wife stealing was so popular, see the Spartan marriage habits. Science rediscovers humanity, but then again there was the habit of killing anyone that didn't fit, and perhaps cooking them for dinner, mankind a strange and wondrous group.
Some people right here in my own house wouldn't exist if it weren't for that peaceful interbreeding.
If it was possible for Neanderthals and homo erectus (or whatever our ancestors were) to interbreed, then, by definition, they were not two different species of human, but two subpopulations of the same species. By definition, speciation has occurred only when the offspring of such a breeding is sterile. In that case, it would not have mattered if Neanderthals and homo erectus interbred, none of the Neanderthal genes would have survived in our population.