The Boers arrived in the Cape of Good Hope to settle from 1652 -- that is 370 years.
When the Boers came, the native people were Khoi-San
Now they may look "black", but genetically they are as far away from Bantu-speaking "black Africans" as those Bantu-speakers are from us.
The Bantu-speakers like Zulus came from west Africa and they expanded when they discovered agriculture (this was completely independent of the discoveries of agriculture in the fertile crescent or in the yangtze valley)
The Bantu-speakers' agriculture was suitable for the tropics with cassava etc. - so their expansion halted at what is now the borders of south africa
the earlier Khoi-San (who had separated thousands of years earlier) were hunter-gatherers so were less dependent on the crop and the Boers came from temperate climates with temperate crops
Zulu expansion came with the adoption of european crops.