The term "ancestors" is used loosely, since physical evidence points toward modern-humans first rising in Africa, not of the Neanderthal line.
And, today no Africans have "Neanderthal DNA" in their genomes.
Non-Africans do have some Neanderthal DNA -- from 2% to 4% -- which is now supposed to have resulted from, ah, er, ahem... Neanderthals in hiding in modern-human wood-piles...
Scandinavian Nanderthal?