You must have forgotten about Carter. I agree with the previous poster, Clinton is #2.
But BillyBob was in the White House twice as long. Hence the No. 1 designation. IMHO.