His age, the political climate, dealing with a war, his age and most important his children will be moving into early teens and he will be moving into older age,
He won't want to miss their growing years for another four years
If Fred serves two terms, his daughter will only be about 14 when he leaves the White House, and their son will only be about 10. Seems to be a good time to do this. His kids will have wonderful memories of growing up in the White House, but will still have their teenage years in their own home, and will be able to live their lives normally, though they’ll have that famous history.