"Show me just what Mohammed brought that was new, and there you will find things only evil and inhuman, such as his command to spread by the sword the faith he preached." - Manuel II Palelologus
Further, it's very desirable that a child have an identifiable mother and father. There's a simple reason that polygyny (multiple wives) is much more widely practiced than polyandry (multiple husbands): even if a man has sex with a dozen wives, there will be no doubt is to the identity of any child's parents; if a woman had sex with a dozen husbands, however, there would be no way of knowing which children were fathered by whom.
Incidentally, the rules about adultery are asymmetric for essentially that reason. If a married woman sleeps with a man not her husband, she prevents her husband from knowing which children are his. By contrast, a married man who sleeps with an unmarried woman in no way prevents his wife from knowing which children are hers.