There is reasonable evidence that the Lemba people are genetically related to Jews, although thyey look quite African.
I think it matters, after a population shift, who mates with whom. Native men probably seldom mate with invader women, while invader men probably routinely mate with native women.
Over time, this will get the "Y" lineage and the mtDNA out of sync.