No need to fear what you will come back to. You should fear trees and stuff like that falling on the house with you in it.
I know you're right-it's just weird.