The main issue for me is the simple ethical one of just knowing what is right and what is wrong and having the gonads to do the right thing in life.
Take freedom to the furthest extent and you have anarchy. Another issue is that illegal drug use is corruption of the innocent by way of ignorant new consumption due to a lack of education or irresponsibility of youth.
Why would you want to grow illegal drug users? Seems so unethical to me. If you make the claim you don't want that, then why push for something that would directly cause new addicts? That would be throwing human reason out of the window. Which illegal drugs can make a person do by the way!
And this raises a whole other issue which I haven't seen addressed by those who advocate legalization. Since our whole form of self-governance is based on rationality, the ability to offer consent, and clear-headedness, why would anyone advocate the "right" to abrogate self-governance unless they were haters of liberty or were just sorely ill-informed? That is why I associate Libertarianism with the Baathist Party of Iraq, which applauds the efforts of the L.P. Do people have a "right" to destroy their own ability to self-govern? If the inalienable right to life/liberty/pursuit of happiness can be self-abrogated (as someone here claimed) then they would have no problems whatsoever with the consensual burning of the Bill of Rights. And who but an enemy of the Constitution (or the ill-informed) would approve of that?