Sumner was the victim.
While he was the recipient of the caning, I do not believe he deserves true victim status. He had insulted a fellow senator, insinuating that he had a slave mistress, and ridiculing him for his inability to speak clearly(the senator in question had suffered a stroke). As the offended senator in question was in no shape to defend his own honour, his cousin took matters into his own hands.
The speech provoking the caning, not the caning itself, marks an end to civil discourse.