Speer was pretty bad. He knew all about the Holocaust, for one thing, though he lied and said he didn't. People who were less guilty than he was got the death penalty.
In many respects, Speer was the cleverest defendant at the Nuremberg trials. He admitted to just enough guilt to gain sympathy and seem remorseful, but not to anything more.
I find it interesting that after the war the Soviets agreed to early release for some of Nuremberg criminals, but they demanded that Speer (and Hess) do their full sentences.