I suspect that if there is any N-DNA in us, it won't be the Y chromosome. If there was a really, really ugly subspecies of human around I doubt a woman could overcome the "ick factor" (and pass on Neanderthal male chromosomes to her children). Men are different though. Most of us know horny guys who'll go after anything that remotely looks female. Consider the bestiality factor, it's usually male perverts who are arrested for sex with animals.
Any off-spring of those "2 am" trysts would then have human paternal-chromosomes and Neanderthal maternal chromosomes My advice to molecular biologists in search of that possibility would be to study the maternal DNA for Neanderthal genes.
Good point, lad. Check the mDNA, indeed. So, my brow ridges came from my mum??