If it was so well known as you say, then how could it be that his oldest daughter said that she never saw her father abuse her mother or them?
It’s easy to say anything about a man especially after he’s dead & unable to defend himself.