There are records of the rumor going back to 1802. In 1811, a school headmaster who visited Jefferson wrote in his diary that day, "The story of black Sal is no farceThat he cohabits with her and has a number of children by her is a sacred truthand the worst of it is he keeps the same children slavesan unnatural crime which is very common in these parts."
Hadn’t heard of that diary entry. Sally was apparently the half-sister of Jefferson long-dead wife. I think all of her children were freed in his will, but not Sally herself.