It’s a trade-off. When I carry a 1911 I also generally carry it with an empty chamber. I also train that way and have gotten proficient at the draw-rack-aim drill. There is time between drawing and aiming, too... and this time can be used to rack the slide on the way up. I just don’t like carrying a pistol with the hammer back on a live round. I’ve decided that for me, it’s an acceptable trade for the significant reduction in risk. I never use the safety switch. Ever.
However... when I carry my Sig, I carry it with a round in the chamber and the hammer down. It has a long double-action trigger for the first round. In this way, it’s not any different than carrying a revolver.
Who would carry a revolver with the hammer back, in a holster? Nobody. Well... nobody ~smart~ anyway. A safety switch just isn’t enough insurance, either. :-)
You should practice cycling the safety off. Rest your thumb on top of the safety, it cycles OFF quickly and easily, but takes a deliberate act. Try carrying cocked and locked (over an empty chamber if you want), to build trust in the safety. It won't cost you much time to drop the safety on your way to racking the first round into place, and racking is significantly easier when you aren't also cocking the hammer.