The point about Madison is that he believed we needed government because men are not angels. You may not realize it but the drugs you are keen for were not available when Madison was writing.
Marijuana and opiates have been known for millennia - in fact, Ben Franklin was one of many who used a tincture of opium in alcohol (laudanum).
You probably don't realize it but most Americans would never believe that the Founding Fathers would have been keen for legalizing pot or the other drugs.
What I realize is that the Founding Fathers put no federal authority in the Constitution to regulate the intrastate growing, distributing, selling, buying, or using of any drug - which makes moot whatever "most Americans would never believe" about it.
historically and presently most Americans do not share your pro-pot views.
Nationwide "54% favor marijuana legalization": http://www.people-press.org/2014/04/02/americas-new-drug-policy-landscape/
I know you say you're personally opposed to pot but in favor of everyone using it if they want, but that's just more sophistry, like the Democrat claptrap on abortion.
Abortion violates a nonconsenting person's right to life - drug use violates nobody's rights.
How do you believe that our society copes well with alcohol yet you've lost a family member to it?
Society goes on without him; if we ban everything with which people might choose to harm themselves, what will be left?
The harm done by drugs is not just to the poor, weak-minded druggie. The harm is to the rest of us as well. We're right not to want it legal as we're forced to bear the burden of drugs and its consequences.