Something similar happened to an acquaintance of my family. He was involved in a minor fender bender with someone angry and not very tightly wrapped. The acquaintance had a concealed hangun on him at the time and at some point in the incident the other person noticed the firearm (I think with all the confusion and fuss, the man's shirt rode up for a moment over the revolver grip protruding from the IWB holster).
So the angry motorist decided to tell the cops he was "threatened with a gun" by this guy. It was ugly for awhile, but fortunately it eventually got smoothed over (after a period of some uncertain weeks). Fortunately, there are usually witnesses to fender benders. If this had occurred in a situation without corroborating witnesses, it could easily have resulted in arrest and possible conviction.
Moral: you shouldn't HAVE to hide your gun in an ideal society, but it ain't an ideal society. There are just too many total butt holes out there just looking to make your life a living hell.