You are correct - 100 mA can stop your heart and kill you. But most deaths are not caused from the shock - they result from the impact with the ground when you fall off the ladder.
I'm a retired commercial maintenance mechanic with a strong Electrical wiring background. You're right ladders likely do get more people and one nearly got me. I've been retired since 1994 on medical issues. But I decided to change out a flood light fixture. I won't work anything hot unless there is no other way so I killed the circuit breaker to the light and rechecked for power. Circuit was dead.
I went up on a step ladder and was assembling the new fixture trying to get the screws to line up with the box. I got hit so hard it knocked me off the ladder and into a banister on my deck. I laid there a couple minutes trying to get my breath and went inside. My wife yelled "what happened it sounded like an elephant fell down". I said I was knocked off the ladder. {she couldn't come check on me due to her disability} I gathered my wits and went back to the breaker panel and the breaker was off. I climbed back up and checked the wires and had no power showing. I grabbed the fixture again and felt a lesser hit and I let go.
I stood there think WTH is wrong here then I saw the culprit. I had goats for clearing brush and I had ran an electric fence to a lower lot below the house I had forgotten about. I was getting my arm into the 5000 volt surger while holding the grounded fixture LOL. The surgers amperage was way too low to kill me but the hit while up on a ladder about did it. Same thing with Yellow jackets and Wasp if up on a ladder the fall will do more than they will to ya.