The definition of what harms another individual is famously very elusive.
Running a meth lab next door to my property may not deprive me of my use of my property directly, nor is it the same as directly assaulting my person physically.
But I think that for the good of myself and my neighbors your purported "right" to run a meth lab on your property should be restricted by law.
Which speaks to a larger principle: each person's actions do not have just one isolated consequence each time they act. They often have multiple consequences.
The historical reality is that the less internal self-control people have, the more external control the state will exert over them.
I prefer a system where the laws are few in number, limited in scope, and strictly enforced.
Of course, such a society only works when citizens behave like adults rather than overgrown children.
Congratulations.