I've also heard rank speculation that there was something else going on between Joe and his daughter.
And it's not inconsistent with what's actually known.
Please, there is SO MUCH dirt on the Kennedys, verified and published, that there's not even a need for rank speculation. That family is like "Dallas" and "Dynasty" had never gone off the air! It's like....well, it's like having the guilty pleasure of reading "The National Enquirer" without having been seen buying it in the checkout lane!
"something else going on between Joe and his daughter..."
-------see the 1974 film Chinatown, with the great John Huston as Noah Cross---could have been a model for Joe Kennedy,