You have a semi-valid point. If the intoxicant has no valid use, then its use is immoral.
Using pot simply to get high is immoral. Using alcohol simply to get a buzz is immoral. Neither is as immoral as murder, except on the ground that even the tiniest sin separates us from G-d and requires the blood of Jesus. Also, when we stand before Christ for judgement I don't believe He is going to accept, "Well, yeah, I did that, but I'm not as bad as Charles Manson."
However, wine is mentioned in the Bible as having medicinal value which has been confirmed by modern medical science. Apparently a glass a day is actually good for you. Whether you can include beer or whisky I can't say as I have not heard the study extended to those beverages. They are not mentioned in the Bible, but then I don't think the Jews brewed beer or distilled alcohol. I could be wrong.
IMO: Jesus taught that all morality boils down to a heart issue. What is in your heart means much more than what passes your lips. One key indicator as to whether a substance is immoral, whether it is pot, tobacco, or french fries, would be the reaction of a person to the phrase, "Maybe you should give that up."
A reasoned reaction indicates that there may not be immorality involved, or if there is the person is open to the prompting of the Holy Spirit to a change of behavior. A knee-jerk, "when they pry it from my cold, dead hands," reaction would indicated there is likely immorality involved.
Where is your heart on pot?
Shalom.