For what he said, yes. Insinuating that an elderly member of the senate is having sex with his own slaves, at an assembly in front of Congress, definitely deserves a good beating.
And at a time when miscegenation was not only illegal, but considered repugnant and immoral, doubly so.
Where in his speech did Sumner make that accusation against Butler? I can’t find it.
He didn't say that actually. What he said was more abstract and figural, rather than literal:
The senator from South Carolina has read many books of chivalry, and believes himself a chivalrous knight with sentiments of honor and courage. Of course he has chosen a mistress to whom he has made his vows, and who, though ugly to others, is always lovely to him; though polluted in the sight of the world, is chaste in his sight—I mean the harlot, Slavery. For her, his tongue is always profuse in words. Let her be impeached in character, or any proposition made to shut her out from the extension of her wantonness, and no extravagance of manner or hardihood of assertion is then too great for this senator. The frenzy of Don Quixote, in behalf of his wench, Dulcinea del Toboso, is all surpassed.
Sumner didn't say that Butler actually was having sexual relations with his slaves. Worse, though was that Sumner mocked Butler's disability.
[He] touches nothing which he does not disfigure with error, sometimes of principle, sometimes of fact. He cannot open his mouth, but out there flies a blunder.
This was after Butler had had a stroke and couldn't speak well.
Butler made the same sort of assertions that you believe Sumner did and he escaped a beating:
According to Manisha Sinha, Sumner had been ridiculed and insulted by both Douglas and Butler for his opposition to the Fugitive Slave Law and the Kansas-Nebraska Act, with Butler crudely race-baiting Sumner by making sexual allusions to black women, like many slaveholders who accused abolitionists of promoting interracial marriage.
If Sumner had actually accused Butler of having sexual relations with his slaves, he might not have been wrong. That was certainly the case for Butler's successor, James Hammond. More details, if you dare, are here.
As senators warned at the time, Sumner did go too far, but just where the line wasn't exactly clear at the time, I doubt he deserved getting beaten to the point where he was incapacitated for over a year.
And we certainly know that plantation owners never slept with their property, right?
Right.
To be doing something and have someone call you out on it is something we refer to as “speaking truth to power”. It is NOT a reason to beat anyone half to death.
It should have landed Brooks in prison, but it proves that “justice for me, but not for thee” was as rampant then as now.