#3 --- 1849 - Zachary Taylor refuses to be sworn in office as 12th President of the USA on a Sabbath (Sunday). Consequently the office of President of the United States of America is vacant for a single day. Urban legend instead holds that David Rice Atchison, President pro tempore of the United States Senate was President de jure for a single day.
But he died in office, so no even four years. Good guess, though - I thought that too.
There is, btw, no prohibition against observing Inauguration Day on a Sunday; General Taylor simply refused to participate on that day.