There is such a thing as pocket carry which sounds like what you need.
Here is a site for just such a holster:
http://www.pocketholsters.com/
Since your attire is so informal, you might also consider buying some shirts designed to wear outside your pants. Bulky enough, then you could go for an Inside Waistband holster or a OWB pancake style.
Ever thought of an open carry fixed blade knife on the waistband? I've go a KaBar TDI, it holsters and draws like a firearm.