The only real barrier these days is employment-based drug testing.
And if we legalized drugs that would still go on and employers could discriminate against drug users just like they are allowed to discriminate against people who use tobacco, a legal drug.
I am not for legalizing all drugs though. I am strongly opposed to legalizing anything but marijuana.