IMHO, it has to do with the existence of a steady food supply, population explosion, and a quite understandable narrowing of genetic variation in later populations. Obviously the largest families tend to have the most descendants. Standing armies were among the consequences of the agricultural revolution(s), and neighboring groups would lose females to the more prosperous settled groups, particularly if they tried a raid against a settlement with a standing army. Harem-style partnerships are very very old, after all. Thanks SteveH.
So why am I thinking that propinquity also plays here?
Something like “While the hunter is away, the farmer comes in to play!”