Why do you think he took advantage of her? It seems like a relationship that was advantageous to both of them.
For me, it doesn’t “destroy” his reputation. He was simply a man of his times. It was what it was.
I’d think worse of him if it turned out he was letting his brother come up on his plantation and rape his wife’s little sister willy-nilly.
Your Oprah-esque pop culture sensibilities do not apply to 18th and 19th century Virginia.