Direct effects, for one. The CSA explicitly notes various aspects of the trade in illicit drugs which cross state lines.
"The subject to which the power is next applied is to commerce among the several states. The word among means intermingled with. A thing which is among others is intermingled with them. Commerce among the states cannot stop at the external boundary line of each state, but may be introduced into the interior. It is not intended to say that these words comprehend that commerce which is completely internal, which is carried on between man and man in a state, or between different parts of the same state, and which does not extend to or affect other states. Such a power would be inconvenient and is certainly unnecessary. Comprehensive as the word among is, it may very properly be restricted to that commerce which concerns more states than one."Now, have you got anything to support your position? Anything at all?
-- Gibbons v. Ogden (1824)