And I should do nothing to interfere with your way!...an armed society is a polite society.
I've seen many pistoleros who could do things with a .32 or even a .25, that others could not accomplish with a .40 or .45. It all depends on mission requirements, and the individual's willingness to maximize one's proficiency with the weapon.
Personally, I carry a large enough frame that I can conceal my .40 Colt, even in casual clothes. In a sports jacket, I can pack my Desert Eagle in a custom shoulder holster. These are the weapons I train and practice with habitually, because I know they are the ones I'll have with me if needed. I never answer the door without one of the two tucked in the back of my belt.
When it gets cold enough to wear a jacket, I will probably carry my Sig Model 239 in .40 Cal. I like to go lighter when I don't wear a sweater or jacket.
...and the .22 in your pocket kicks butt on the dozen .45's you left in the safe.