If they're lucky, that's the only downside. It's usually more sinister.
In most polygamous groups, "excess" young men are simply killed, or exiled from the group, to reduce competition.
Polygamy is hell on males, except for the select few.
Multiple Mother's in Law!!