Harem were composed of infidel women (mostly slaves) lorded over by muslim women.
And those slaves were caught and sold in the only viable and lucrative business of the times by Christians and Jews; the dark time no one wants to talk about and for which many westerners are humiliated to dwell on.
See here for further explaination:
Why We Are Afraid, A 1400 Year Secret, by Dr Bill Warner
https://www.youtube.com/watch?v=t_Qpy0mXg8Y