His wife died while he was working on the Manhattan Project. I think that pretty much messed him up. I'd heard about him hitting on the wives of grad students, but not much else.
Reminds me of the story of Hawking, who had a long affair with one of his nurses when he was still able to get about. When his ex-wife was interviewed, she said, "He's not bleeding Einstein, you know."