The Dept of Defense is not necessarily a Constitutional agency -- in fact, it could be easily argued that it is not: the Constitution only allows for the Army and Navy (Marines are a sub-department of the Navy), which means that the Air Force is extra-constitutional. Thus the DoD cannot be said to be wholly constitutional.
It could be argued that the Department of Transportation is more Constitutional: this is because the Congress is supposed to make commerce between the States regular, which is much of what the DOT does (though it could be pruned very much, but that is like most government agencies in general); the Post Office is another agency, being explicitly mentioned in the Constitution, which is more Constitutional than the DoD.
I think that is a bit of a stretch.
The founders clearly intended for the nation's common Defense - so much so that it is in the preamble.
The argument you advanced regarding commerce has been utilized by every tyrant-wannabe, Socialist, and Communist in US Congressional history as an excuse to impinge on American freedoms, from everything from gun ownership to health care.