For the Neanderthal side that's a serious problem since their remains would have gone "under the ice" numerous times and only a small sample of their quite lengthy history can ever be known.
What we don't know is whether or not any of the Neanderthal line returned to Africa ~ e.g. during those 5,000 year long periods every 100,000 years when the Sahara is in a pluvial and filled with trees, grassy plains, and plentiful game.
We don't know if our type of folks actually made it to Europe during any of the interglacials of those times just before they were destroyed by advancing ice sheets.
At the moment we only have a good grasp of our own line of folks coming from Africa and staying over a major glacial sheet advance by residing in South Central Asia.
There are entirely too many possible points of Neanderthal and proto-modern man to interact over that 500,000 years for us to exclude anything.