"Pornography doesn't rape people -- people do"
And for clarity:
Businesses don't restrict rights -- people (who own businesses) do.
It's important to remember that the problem may not reside at the local property but with an owner who lives a thousand miles away in Chicago where he supports Mayor Daley and all Daley stands for, and claims an ability to restrict the rights of others in communities around the nation because he owns a few square feet of real estate in each such community.
Governments don't restrict rights -- people do.
I just threw that in because so many conservatives are like anti-gunners who blame guns (the tools) instead of the people who operate the tools. Too many Conservatives blame Government (the tool) instead of the people who operate the tool. Conservatives need to stop blaming the tool, and take it for their own to use as they would any dangerous tool -- safely, as intended, without harm to themselves or others.
Businesses don't restrict rights -- people (who own businesses) do.
No person has the right to enter another person's property. That's the principle that falsifies your premise. No person's rights were restricted. That's all that need be said.
But since you came out to play...
Freedom of association is the root of objective law in society. It takes two people to mutually agree to associate with one another. It takes just one person to refuse to associate with another person -- just say "no". Only when freedom of association is violated does a person have probable need to defend themselves -- such as a business owner defending his business/property against a person that violates store policy/rules.
Ignorance proclaims that an equal man at home becomes a second-class citizen when he goes to his store/business. Or does it include that a person in their home can't make his own gun rules/policy?
Governments don't restrict rights -- people do.
I often use the phrase, "politicians and bureaucrats" rather than "government", when making reference. It differentiates from people. Sometimes I just go with "parasitical elites in government".