Almost all situations are not going to be throwdowns where who clears leather first wins. How long does it take to flick off a safety?
Most gunfights last less than 2 seconds. If I can save .1 second I just got a 5% edge. If I fumble a safety under stress I just gave up .5 sec giving the opponent a 25% edge, maybe more if he has already drawn. I like to play odds more than just gamble on a situation I can’t predict.