So you would have had the baby born alive and a nursing mother who had lost her own child in the last few days with a mate who was willing to allow her to nurse another man's child knowing that she would not give him a child until that one was done nursing. If you had all those events come together then you could have your scenario.
Conclusion: Possible
But it would be stretching things a bit. Might make a good movie script!
Standards of beauty change. Maybe the Neandergals looked pretty hot to Sapman.
Or maybe the Neanderthals were a race of genetic geniuses trying to breed a race of untermenschen to serve them.