By my count— FDR, Nixon, Carter, Reagan, and Clinton each unseated a one termer. I get fuzzy on who was in office prior to the 1930’s
Humphrey was an incumbent Vice President not a President. Wilson unseated Taft in 1912 because of third and fourth party Roosevelt and Eugene Debs.
The last three incumbents who lost, Ford, Carter, and Bush, had serious challengers in the primaries and two of them Carter and Bush, had a serious third party candidate.
Nixon unseated no one. He won an open presidential race in 1968 after one-and-a-half termer LBJ announced his retirement.