Lets assume that he did sire children with her (which I seriously doubt in that there were some very nice looking white women around in that day). Did it improve their gene pool? You can pretty much take that for granted and they should be thanking Jefferson and not criticizing him.
Remember that next time your spouse cheats on you.