You argument might work if we weren't living in a welfare state. You claim it's not going to harm me. But as it stands now, the government will take my hard earned money to give to the pot users. And you know it will. Don't claim legalization is libertarian in our world. How is taking money from me to support pot users and their administration libertarian. It's not, it's socialist, or worse.
Everybody gets victimized in some manner or another. Everybody certainly feels victimized at some point or another.