Yes, that's the rumor.....however, Cassini has a lot of credibility about bedding down in and out of marriage, due to the fact that Oleg was one of the foremost swordsman who plied his wares on two continents.
In Hollywood (as Gene Tierney's husband), New York, where he was a designer, and the entire European continent which was his second home and his winter ski playground, Cassini knew all the notorious womanizers like Rubirosa, and traveled with the best of the them all his life.
More than a rumor: there were wiretaps on some of that. For a while JFK was afraid it was going to mushroom into a US counterpart to the Profumo-Keeler scandal, but Congress was persuaded not to make an issue of it because there was enough dirt to bring both parties down, so it was hushed up.