Yes, punching someone in the nose is a violation of their rights, but it's also a simplistic example to use.
Drug use, entirely apart from the drug war and all it entails, DOES affect others. Ask anyone who has an addict in the family to tell you honestly if it does.
Similarly, unrestricted immigration DOES negatively affect others. Just one example is California's hospital system, which has only 1/3 the number of ERs per capita of the national average.
Just because an action doesn't have an immediate, obvious negative impact, it isn't necessarily sunshine and rainbows. "No man is an island" is still true. There are many consequences in life which are profoundly serious, but not as obvious as a bloody nose.
So does anything taken to extreme. Should we really outlaw certain foods because some people cannot control their intake?