Your citing the Notes on Virginia was timely. Just last week I purchased an old copy of "The Life and Selected Writings for Thomas Jefferson" from a wonderful used bookstore in Denver. It's almost 800 pages.
Your post took me to this:
Back in the late eighteenth century the Virginia slaveowners who were Jefferson's contemporaries hadn't taken this Jeffersonian antislavery seriously. They knew Jefferson personally, and knew he meant no harm. And many of them were in the habit of saying the same sorts of things themselves, in appropriate company.
By the mid nineteenth century, however, southerners had to take Jefferson's antislavery writings seriously, because northerners were taking them seriously, and using them against the South. Taking the Declaration of Independence in conjunction with Jefferson's antislavery utterances (well publicized in the North for more than two decades), northerners were able on the eve of the Civil War to read antislavery intentions into the Declaration of Independence itself, and thus to enlist both the Declaration and its author on their side in the coming war. In a letter of April, 1859, Lincoln wrote,
"All honor to Jefferson -- to the man who, in the concrete pressure of a struggle for national independence by a single people, had the coolness, forecast, and capacity to introduce into a merely revolutionary document, an abstract truth, and so to embalm it there, that today, and in all coming days, it shall be a rebuke and a stumbling block to the very harbingers of re-appearing tyranny and oppression."
Source and more...
It’s true that many of the Founding Fathers are condemned in some quarters for being racists and slaveholders, but the fact remains that when they came, one by one, onto the world scene in the last half of the Eighteenth Century, slavery was endemic throughout the colonies with scarce a whisper of opposition to be found anywhere, and that by the time they departed that same way, one by one, in the first quarter of the Nineteenth Century, slavery had been reduced to a despised and crippled institution, confined to one segment of American society, with a raging abolitionist feeling that had even penetrated the South. And, it was the sentiments and the philosophy of the Founding Fathers which fostered that dramatic change.