That's the point: if his rights are properly understood, there is no limitation to them. He doesn't have the right to initiate aggression, and therefore he can't molest you, steal your stuff, vandalise or what have you. If his actions affect nobody's person or property except his own, then anything goes. Or, if all affected parties are consenting, then again anything goes. Those are exactly his rights, and there are no exceptions of any kind.
The only one who can impose limitations on a man's rights is the man himself. He does that by making contracts, whether express or implied. If he marries, he's making a contract whereby his spouse can live in his house without being called a trespasser. If he takes a job, he's making a contract whereby another person can suddenly tell him when to get up in the morning, or what to wear, or whatever else the contract might entail. In our dealings with others we make implicit contracts constantly. The McDonalds, by displaying an "open" sign, offers a contract that we can enter and eat unmolested. By entering, we accept the contract, and in return contract that we'll line up, pay for our food, eat it without bothering the other patrons, and bus our own trays.
What’s in dispute is where the property owner’s rights end and the rights of anyone else begin.
All of our rights begin and end with property and contract. We own ourselves and our stuff, and we can do anything we want to our selves and our stuff, but we can't do anything whatsoever to anyone else's things or stuff without their permission.
Regarding your first paragraph: That's what's in dispute. Even if you and I totally agreed the paragraph was correct, it would still be in dispute because not everyone believes it to be correct.
Your first paragraph is fine as a statement on your part but it doesn't end any dispute as far as I can see. I see no reason for anyone else to believe it is a true statement as opposed to something you erroniously (from another's view) believe to be true. (Please note that in the preceding I did not dispute your statement. I'm not saying I never will, just that I did not then.)
Now on to something else:
"The only one who can impose limitations on a man's rights is the man himself. He does that by making contracts, whether express or implied." "In our dealings with others we make implicit contracts constantly. "
In other words:
A Implied Contract is an agreement among people, though it’s not necessarily formal (and may even be hypothetical, reached through a sort of consensus). In short, it’s an agreement about what rules to follow in pursuit of whatever goals the people have in mind that they wish to pursue jointly.