Some years back, a friend of mine shot at a woodchuck from her rear deck. Given that she was shooting away from the house, I'm not quite clear how her action would have posed any danger whatsoever to anyone inside.
Probably no danger as long as she wasn't shooting toward another building and knew that all occupants of the house were in the house or otherwise accounted for.
I'll shoot WITHIN my own house in self-defense. With hunting, it's a discretionary shot, and if you're sure you're not going to hit something you wish you hadn't, pull the trigger.