More than a rumor: there were wiretaps on some of that. For a while JFK was afraid it was going to mushroom into a US counterpart to the Profumo-Keeler scandal, but Congress was persuaded not to make an issue of it because there was enough dirt to bring both parties down, so it was hushed up.
Referring to a July 1965 FBI document that claimed JFK was apparently involved in sex parties at the Hotel Carlyle in New York City?
Maybe Cassini was covering up for JFK, although Oleg dished the dirt on nearly everybody else, including himself.
Don't want to appear to be defending JFK......just passing along info that's hardly been mentioned, from what I consider to be two reliable sources......Cassini and Frank's valet.