Any time a government bans something and there's a demand for the banned product or service, a black market will sprout up to fill that demand. That's basic Econ 101.
So, using some of that logic, we should open the borders to everyone and anyone to prevent illegals from coming through? (Not that they aren't open now...) Sure, they wouldn't be "illegal" any more, but there would be such a surge in numbers we couldn't handle it. (Not that we can handle it now...)
Laws are the only things that keep some people from using drugs. Take the laws away, and you'll see a lot more drug abuse, more access to drugs by children in their own homes, etc. I don't want to clean up that mess.
The criminals will find new high-priced black markets and human degradation to deal in. If they have to sell pot or whatever at reduced legal market prices, they won't be making the money they are now.