Didn’t Dolley Madison die in near-poverty also (albeit due to her ne’er-do-well son’s spending habits, but still...)
Harry Truman was pretty poor when he retired, had to live in his Mother-in-laws house.