I've said this on FR ever since I started participating in the Drug War Threads. There is no harm unless it's proveable in court.
Think of the prohibition of smoking in business establishments. Think of the business owner that chooses not to allow certain acts or people on their property. Assuming the business owner or property owner is not holding a person hostage the employee or customer or home-visitor has the choice to not associate -- leave the property -- as much as the business owner or home owner has the right to refuse to associate.
The status quo claim is that it violates a person's rights for a business to refuse a person access to the business. Yet, to pass a law forcing the business to grant access to a person violates the business owner's rights.
The status quo inverts justice.
Then by your "reasoning", there's no harm in the drug laws.