Both a buddy of mine and myself have prominent brow ridges and often joke about the Neandertal in our blood. And, he's Scots by birth and I'm Scots by Y-chromosome...hmmmm.
I suspect that if there is any N-DNA in us, it won't be the Y chromosome. If there was a really, really ugly subspecies of human around I doubt a woman could overcome the "ick factor" (and pass on Neanderthal male chromosomes to her children). Men are different though. Most of us know horny guys who'll go after anything that remotely looks female. Consider the bestiality factor, it's usually male perverts who are arrested for sex with animals.
Any off-spring of those "2 am" trysts would then have human paternal-chromosomes and Neanderthal maternal chromosomes My advice to molecular biologists in search of that possibility would be to study the maternal DNA for Neanderthal genes.