Except for wars. With all the taking of captives for slaves and raping and pillaging, there was some room for genetic diversity.
You're right. I forgot. Huge slave trade esp. in Roman times.