That actually did happen over the period from the 8th Century through the 18th Century, as North African Muslims (the Barbary Pirates and others) conducted slave raids all along the northern Mediterranean, with around a million Europeans captured and enslaved
Now, if the Game of Thrones writers wanted to deal with REAL history, they could have a series about African Muslim slave raiders and their white slaves. But that would be too politically incorrect.
Muslims had been pulling about a thousand slaves a month out of Africa for about a thousand years. Males would be castrated. Females were sex slaves. Many marched across the Sahara where 90% slave losses ocurred. They left them as they dropped.
African slaves were sold to China from the east coast of Africa.
I recall Mary Shelly mentioning the problem in her novel, Frankenstein.