"Let say I invite someone into my house to do the housekeeping, cooking, etc. "
Fine.
Then it's your problem, not mine - as long as they stay in Your house as invited and not wander (unasked/uninvited) into mine.
Enjoy your freeloaders.
Let's say I come over to your house with my army, and force you off your land. Then I build a bunch of houses for myself, my army, and other people like me. Then you come back and instead of trying to take your land back, you just want to work, and make a small living for yourself and your family.
Too bad, I stole this land fair and square.