Well, Phoenicians have lived in North Africa for several millennia by now -- a period of time comparable to that from the initial alleged exodus. How come no poeple seem to develop Negroid features because it lives in Africa after migrating from elsewhere?
There is reasonable evidence that the Lemba people are genetically related to Jews, although thyey look quite African.
I think it matters, after a population shift, who mates with whom. Native men probably seldom mate with invader women, while invader men probably routinely mate with native women.
Over time, this will get the "Y" lineage and the mtDNA out of sync.
"Several millennia" is just a speck of time compared to 80,000 years. And "environment" is not just geographical location. Environment is all of the factors that existed at a certain period of time: desease, weather, air quality, competition with other species, etc., etc., etc.,....
If Africa had been empty of people, and a group from say, Europe or America migrated there in our current time, there would be no way to predict how they would turn out 80,000 years from now. There would be just too many factors to consider in their environment.