Biologist Alan Templeton of Washington University St. Louis has found hints that some people of European ancestry carry genes that emerged in Europe more than 300,000 years ago - far before our main ancestors left Africa. There's some speculation that genes associated with light skin and red hair first arose in Neanderthals, for example.
Both a buddy of mine and myself have prominent brow ridges and often joke about the Neandertal in our blood. And, he's Scots by birth and I'm Scots by Y-chromosome...hmmmm.
Unless the light skin and occasional red hair of our ape ancestors emerged again when dark skin no longer gave a reproductive advantage due to living in an environment with less sun than Africa?? The argument of course being that as human ancestors lost their body hair covering, dark skin evolved as protection for the skin from the sun -- in the Northern climes, light skin giving more sun exposure provided a reproductive advantage in a environment with limited sun. Always an argument one way or the other, isn't there.
Actually, I am inclined to believe, like you, that a combination of factors, one of which was inter breeding, contributed to the "disappearance" of the Neanderthal.