The infant mortality rates impacted longevity, but in a nuanced way. If average lifespan was, say, 40, anyone who survived beyond infancy actually had as much chance of living to 70 as much later. The high infant mortality rates skewed longevity, which was an average.
You make an excellent point from the actuarial perspective on longevity. It especially applies to my grandmother’s progeny with youngest adult passing at 53 and the oldest at 94, most skewed into their 80s at their passing. The one child death was at age 4.