That way any "possession" or "use" of your property by those folks never becomes an "adverse possession".
Get it in writing too ~ file it with your deed down at the courthouse.
Alternatively you can put in a barrier, but folks have done so and lost in court. Giving permission and proving it has never lost ~
A variation is to get them agree to pay rent for crossing your land.