Besides larger family sizes, and polygamy, better diets led to lower infant mortality. Anyone who has done genealogy finds both huge family sizes (often finding a single family more than once, depending on how many generations one looks back) as well as perilously narrow family lines (no first cousins, parent with no first cousins, grandparent on same line with no first cousins).
And, in my best heads I win tails you lose fashion, even if there were no change in infant mortality, polygamy would drive up the survival -- and hence lower genetic diversity among descendants -- of the the early farmer dad's double helix.
Added to that is maybe this:
War in the ancient times was an all male activity with rare exceptions. Wiping out all the males in an AO, or reducing surviving males to slave status was fairly common, iirc. And females all went to the victors.
I suspect that the world was awash in small scale wars back then. Village against village, region against region, and such like. And the wandering hunter/gatherer purist clans that refused to make the transition to settled agri life working as raiders on those settled villages.
So, imo, it was probably a whole host of stuff that added up to what we see in the gene record.
7700 years ago marks the MASSIVE eruption of Mt Mazuma - credited with millions of TONS of ash deposited across the northern hemisphere. Could have had an effect on population, but why men more than women? But Egypt built their pyramids soon after - surely in only 1200 years from this event 7700 years ago, they’d not have to rebuild a civilization.