Nothing shouts "I have premature male pattern baldness" as loudly as a 19 year old college student whining about having to take his hat off.
And nothing smarts worse than a sun burn to the top of the bare head where hair is no more or real thin :>} I started loosing the top of mine about 20 years ago. I wear one {cowboy style} and have for several years. I'm 53. It's functional in many ways including protecting a $5000 investment handing on the top of my ears from the elements.
I'll take it off in a church, court room, funeral, or and other such events of serious nature like the National Anthem etc. That's about it. I walk with a cane a lot of times so taking it off would tie up use of both hands. I don't see the fuss about when and where to wear a hat except in places like church, court, funeral. and a movie house for obvious common sense reasons that's about it.
On the other hand unless I go into stores I shop regular and know the clerks or whoever I'm doing business with I'll say yes mam and no mam in answering them. It's not how you look that matters so much in public as to how you treat others.