Sometimes wifi is intended for the receipt of others (e.g., starbucks, or my neigbor's "guest network"). If you are broadcasting an unencrypted signal, then I don't see how your analogy of going into someone's house fits - there is no intrusion.
It is more like they were taking pictures of the street, and you left your door open and they accidentally captured you naked as you were walking to the shower.