Although Neanderthals, at first sight so similar to modern humans, were once placed on the last rung before mankind, fossil DNA hints that they may not even be on the same ladder. Their mitrochondrial genes are quite distinct from our own. They were not the ancestors of human genes but followed a separate path. For mitrochondria at least, Neanderthals and ourselves split half a million years ago.
Mitochondrial DNA is only passed on by mothers.
Nothing saying a neanderthal couldn't be someone's babydaddy...