I do that because that's where the people passing the laws claim they find the authority, and they leverage that to claim authority far beyond. That you consider it "Defense" doesn't change that. It is what it is, and that's what we have to deal with - not what it would be in your perfect world.
I argue that they only cite the Commerce clause because they are lazy, and have become accustomed to citing it so often to justify anything which they do. I argue that the proper clause of the U.S. Constitution to cite is Section 8 dealing with Defense of the nation.
My point is, the U.S. Constitution does grant authority to interdict any material which may be regarded as being used in an attack upon the nation. Nerve Agents, Biological Warfare Agents, Nuclear Fissile Material, and Drugs all Constitute Legally prohibited Contraband in Lawful accordance with the Articles and Requirements of our U.S. Constitution.
"Commerce" only applies to lawful exchanges of goods and services.