Most of the time I carry a 380 ACP or 9mm. In special situations I will carry a .357 Mag or if in the deep woods a .44 Rem Mag.
I have no concerns carrying a .380 ACP with a spare magazine full of ammo. The .32 AVP and .380 ACP don't get the respect they deserve.
Lucky Gunner recently posted a survey of their results in 22, 25, and 32.