Johnson knew his days were numbered in 1968. There was a strong family history of heart disease and he had already suffered at least one and I think more than one heart attack. Within a year or two he was essentially a cardiac cripple and within a year or two more he was dead.
He also hated RFK more than anything and when McCarthy did well in New Hampshire, I think LBJ saw the writing on the wall. I think he did the political calculations and saw RFK entering the primary (which he did) but of course, did not know that he (RFK) wouldn’t live to see the convention. If you look at politics historically you’ll see that very often when a politician decides to retire, it’s often (not always but often) due to a calculation that they’re going to be tossed out anyway.
I think LBJ’s decision was partly health related, partly based on politics, and partly based on the toll Vietnam had taken on him.
Yet Humphrey who promised to continue Johnosn's policies (and the war) won the nomination and narroly lost to Nixon in the election. LBJ could have easily won the nomination and the election if he had wanted. The convention was still machine dominated then and LBJ controlled the machine.
Why Johnson decided to make that speech in March or April (as I recall) of 1968 where he said he would step down is a matter of conjecture. But some have reported that LBJ was a life-long manic-depressive, and he was in one of his depressive moods then and no one was able to stop him from giving that address.
There have been reports that he damn near did the same thing in 1964 when he was in one of his depressions and only his wife was able to talk him out of it.