Three words: Shelia Rauch Kennedy
Do you honestly think that Joseph Kennedy had ANY intention of being faithful to his wife during their marriage? Old Joe passed on his 'Continental' attitudes about marriage to his sons, who passed it on to THEIR sons. That attitude is that ANY woman is fair game for a Kennedy man, whenever he decides he wants to have her. If he's married at the time, too bad for the wife.
I doubt whether ANY of the Kennedy men has ever entered into a valid Sacramental marriage, as is required by the Church. Any of them, or their wives, would have had grounds to annul any marriage into which they entered.