Spend some time in Amesterdam and tell me if you have the same opinion about legalized drugs. On the criminal law issue, sure the federal government has expanded way beyond what the framers could have imagined, but they did have federal criminal law, even back in the days of Jefferson.
I will say, additionally, I don't think the intent of the commerce clause was to give Congress a hook on which to hang federal law.
However, I think that sometimes you simply have to cede to history--Congress has been doing it for too long to upset the apple cart at this point. I think, however, its powers to define criminal law under the commerce clause should be strict and narrow. No laws against handguns on school grounds or violence against women statutes, for instance.