Not necessarily true w/r/t spouse. In most states, each spouse has an equal right to full enjoyment of the property, so each person can invite someone in irrespective of the wishes of the other. Some states may be different. Of course, if you are in a house and one spouse says "stay" while the other says "get out," the course of prudence is to GET OUT!
I suspect in most cases it is not at issue because both spouses hold title to the property. In the case where the house is titled or rented in one spouse's name, might be different, I guess.