It was still wrong even though he didn't do it.
Not trying to make him look good. I am not the fan I used to be. Jefferson was way too much the fan boy of the French Revolution for my tastes. Just trying to be fair and not just pile on because it’s now fashionable.
Anyway, Jefferson was gone from Monticello through most of that time period. Even when he was there, I doubt he kept tabs on his brother. Another candidate is his brother’s son Isham Randolph Jefferson who was known to be there more often then Randolph.