The “white” South Africans has been there for so long that they do not have country to return (most of them).
Also, most of them do have some black ancestors.
Just like maybe 25% of White Americans do. (see Pope)
There's also the fact that most South African blacks are Bantus who are no more "native" to South Africa than the Afrikaaner whites, and haven't been there much longer. The only "native" South Africans are the hunter-gatherer Koi-San (!Kung) tribes, who were largely displaced by the Bantu herders. So Afrikaaners have every bit as much right to be in South Africa as the Thembus whose ancestors hailed from east Africa.
Mixed race South Africans are called "colored" and the blacks HATE them too!
The Afrikaners came with their OWN white women and children! The ONLY intermarriage was then, if it happened, between the French Huguenots who had been living in the Netherlands and maybe the Dutch. THAT'S IT!
Much later on, their have been some Afrikaners marrying Brit descendants.