My understanding is Brady has gone twice in the past, maybe he just doesn’t find Lord Foul special enough to see him again.
He actually went three times and they were all with President Bush.