The "pot defense" does sound pretty lame. However, this, I agree with:
"Anyone who unlawfully takes a dangerous mind-altering controlled drug and who then finds that his mind is altered in unexpected and undesired ways must take responsibility for his subsequent actions."
The issue is "dangerous". Pot is about the least dangerous drug there is.