Your military id is your passport.
And yet...
How do you think sailors, marines, soldiers travel abroad? they have passports.
Amazing!
“When entering from the United States, U.S. citizens are encouraged to show a U.S. passport. If they do not have a passport, they should be prepared to provide photo ID and proof of U.S. citizenship such as a birth certificate, naturalization certificate, or expired U.S. passport
Read more: http://wiki.answers.com/Q/What_do_US_citizens_need_to_enter_Canada_and_re-enter_the_US#ixzz1zxeLDuDs
Answer is NO you do not need a passport. However, I would imagine you would dig up any excuse not to look into a job anywhere on your own.