With his wife, Jefferson had two daughters who survived childhood. They married relatives who also came from slaveowning planter families. His elder daughter objected to slavery but went on owning slaves.
Slaveowning was a sign of status that it was hard to do without if one aspired to a social position and upper class acceptance. Families that once had slaves often wanted to become slaveowners again, and they could often hope that an enterprising son would restore their fortunes on the frontier. Individuals with misgivings about slavery were often in families and communities that didn't share such doubts so they didn't act on their convictions.
That's what the Hemmings story is like. Jefferson's opponents went around saying, "You know Jefferson is a widower and gets lonely. Why, they say he's actually fallen in love with his slave woman. He even takes her to galas with her dressed to the nine's and introduces her not as his servant, but his girlfriend! Is this who you want running your country? He even takes her abroad and does this!! What a horrible representation of our country! Some say he'd marry her if the state would allow it!!!"
Of course, if those allegations were true then today we'd say that Jefferson was ahead of his time. Me, I don't want to give Jefferson credit for that part because, again, I read years ago that the entirety of the Hemmings story came from Jefferson's political opponents anyway. So I neither shame him for it (today's Jefferson haters go so far as to say he raped her) nor applaud him for it (if as the original rumor tellers say it he was ahead of his time for being color blind in love). The source of the information is questionable at best, flat out lies at worst.