Until a few months after 9/11, passports were not required for travel between the U.S. and Canada. I visited Canada many times with only a birth certificate and drivers license.
So did I. Not many times but at least a dozen or more - and this was long before 9/11
You still can. My mother in law went with my wife to visit my daughters at college in Montreal this fall. I stayed home to get my son to his baseball games. She brought just a drivers license and birth certificate, and had no problem crossing both ways. It is just a lot easier to use a passport. A passport will also make life easier at the airport for domestic flights too.