"just that he had serious qualms about slavery."
I do not believe this sentiment goes far enough. Having spent decades in the public light and not once found promoting slavery legislatively is a little more than "serious qualms".
George Washington had serious qualms about it. He did far less than Jefferson did.
I was talking about Washington, not Jefferson.