Congratulations on an excellent point.
Every person on the planet probably has both slaves and slavers in his ancestry. The only question is how far back you would have to go to find them.
Africa, not very far back at all. . . still going on in many places. American South, not very far either. Central America, not much earlier. In Europe, there were still slaves in Britain in Christian times, in the rest of Europe you have to go back a bit further. But of course there were slaves everywhere the Roman Empire extended, and before that the Greeks and Egyptians.
But yes, it's just a matter of how far back you have to go.