Well, people will gossip about a man marrying his secretary.
Even if they had not worked together in many years, people are still gossiping. Even though they were both married to others, people still gossip.
I had such a thing for her back in the day.
THEY EACH LOST LONG TIME PARTNERS.
LET THEM ENJOY WHAT TIME THEY HAVE LEFT.