[Roscoe]

"But rightful liberty is unobstructed action according to our will within limits drawn around us by the equal rights of others." --Thomas Jefferson.

If society grants you a right to swim in a public pool, others will enjoy that right equally.

[Alan Chapman]

And how are the equal rights of others defined?

[Alan Chapman]

If society grants you a right to swim in a public pool...

You're confusing a right with getting permission to enter private property from the property owner. You don't have to ask permission to exercise a right.