Sally Hemmings was the half-sister of Jefferson's dead wife.
Not only was Sally Hemings his wife’s half sister, she was raised in the Jefferson household to help tend their children. She accompanied his youngest daughter on the voyage to France when Jefferson was there on State business. Relationships between people, long dead, are much more complicated then the judgement of those living.
I wouldn’t doubt she remind him of his wife and home and if she had not been a slave, he would have married her. People back then tended to marry within the common families more often then outside them. One look at the Jefferson and the Wayles family tree shows this.