If the affair has been over for four years, forget it. It means nothing. I really do not think that very many Democrats care about it even if it was yesterday. I would not put it past the plenty of Democrats to vote for a man who intended to openly live with a mistress in the White House. They would probably like it.
I read another article (it's on FR somewhere. There's so many) that reported he recently shipped her overseas. Right before he announced his running. It's a long term affair, I guess.