Off the top of my head I can think of three founding fathers who never served in the military - Jefferson, Adams and Madison. All three served as President.
I just checked and I think you got them all. Monroe served in The Continental Army.