They took chances, they died.
Not my ancestors—although they might have run from the sounds of battle.
;)
If they had died without reproducing you would not be here. :)
Something else is that the Y does tend to pass only through the male line so any man who only had daughters who survived and reproduced would vanish out of the reproductive line while any woman who had only sons would still show up.
So part of this is math and biology.