He didn't say that actually. What he said was more abstract and figural, rather than literal:
The senator from South Carolina has read many books of chivalry, and believes himself a chivalrous knight with sentiments of honor and courage. Of course he has chosen a mistress to whom he has made his vows, and who, though ugly to others, is always lovely to him; though polluted in the sight of the world, is chaste in his sight—I mean the harlot, Slavery. For her, his tongue is always profuse in words. Let her be impeached in character, or any proposition made to shut her out from the extension of her wantonness, and no extravagance of manner or hardihood of assertion is then too great for this senator. The frenzy of Don Quixote, in behalf of his wench, Dulcinea del Toboso, is all surpassed.
Sumner didn't say that Butler actually was having sexual relations with his slaves. Worse, though was that Sumner mocked Butler's disability.
[He] touches nothing which he does not disfigure with error, sometimes of principle, sometimes of fact. He cannot open his mouth, but out there flies a blunder.
This was after Butler had had a stroke and couldn't speak well.
Butler made the same sort of assertions that you believe Sumner did and he escaped a beating:
According to Manisha Sinha, Sumner had been ridiculed and insulted by both Douglas and Butler for his opposition to the Fugitive Slave Law and the Kansas-Nebraska Act, with Butler crudely race-baiting Sumner by making sexual allusions to black women, like many slaveholders who accused abolitionists of promoting interracial marriage.
If Sumner had actually accused Butler of having sexual relations with his slaves, he might not have been wrong. That was certainly the case for Butler's successor, James Hammond. More details, if you dare, are here.
As senators warned at the time, Sumner did go too far, but just where the line wasn't exactly clear at the time, I doubt he deserved getting beaten to the point where he was incapacitated for over a year.
I saw that part of the speech before I replied. Because Sumner used the word mistress as a metaphor for slavery in that passage, I figured either Sumner was misunderstood, or Brooks overreacted, or Brooks was reacting to something else Sumner said.
I, too, see no justification for the beating of Sumner. That beating sounds more like attempted murder.
It might have been seen as all that more offensive if it was in fact true.
As senators warned at the time, Sumner did go too far, but just where the line wasn't exactly clear at the time, I doubt he deserved getting beaten to the point where he was incapacitated for over a year.
Hitting him in the head so many times was excessive, because the injury he suffered was verging on maiming. Of course the people back then didn't take affronts lightly, and I have no doubt some wish he had been killed.
I find myself nowadays wanting to hit leftists in the mouth for their lies. I've reached a point where I don't think arguing with them serves any useful purpose, and allowing them to shriek and carry on and spew lies and insults serves no good purpose either.
Punching them in the mouth might serve to make them keep a civil tongue in their head.
I see parallels between the divisions then and the divisions now.