I have no children in the house, but it is usually on my hip, as it is now, otherwise, very close nearby always in a holster while loaded.
At night, it is on my nightstand, holstered, but loaded.
Another issue is re-chambering ammo over and over. This can cause an un-safe condition referred to as "set-back" resulting in over-pressuring a round. I prefer to keep my weapons loaded until it is time to go the range.
Here is a good site on concealed carry if you have not seen it.
http://concealedcarryforum.com/forum/
Lots of good information there.
Good point.