Actually, my understanding of this debate is that this is still under some dispute. There are a few people who claim that the Neanderthal population was absorbed into ours. I personally doubt it. I tend to agree with you.
Life would be radically different today if the Neanderthals had made it to the present time. There's a very good book out there called Extinct Humans about the evolution of our species. It points out that speaking in terms of evolution, there have been many different homo sapien-like lines that evolved. All for one reason or the other went by the wayside. Very interesting book that changed my whole perspective."we are alone in the universe" bunk would've never even developed. Since they didn't make it, probably due to our ancestor's crowding them out, we now live in a world where nuts such as Mr. bin Laden think they're God's gifts to the universe........
I read this book and thought it was okay. It pointed out that different species of humans were living at the same time. I just don't think the Neanderthals were different species.