Letterman was actually quoting Wilson Mizner who was a manager of The Brown Derby restaurant as well as a playwright.
http://en.wikiquote.org/wiki/Wilson_Mizner
good catch. I knew I had heard that somewhere before