According to the code, you only challenge your social equal. Sumner didn’t qualify.
Then why haul out the Code Duello to begin with? Sumner came from a prominent family, was educated at Boston Latin, Harvard, and Harvard law, and was fluent in several languages. Brooks was a wastrel, an unsuccessful duelist, and a mediocre politician. Sumner's intellect and sophistication put Brooks to shame. Brooks was the low-life, not Sumner. He beat an older, unarmed man almost to death, depending on he fellow lunatic, Lawrence Keitt, to hold off any interference with a pistol. Hardly the acts of gentlemen. More like street-level thugs.
In the end I suppose Sumner had his revenge. Brooks went on to his early and unlamented death in 1857. His victim lived for almost 20 more years and died a respected man. Brooks was planted in some foresaken spot, God knows where. Sumner was was awarded the honor of lying in state in the Capitol before burial at Mount Auburn in Massachusetts. Brooks is a trivia question and Sumner's memory lives on in the number of counties and people who were named after him.