Wasn't he asked to leave as Speaker of the House? He must have been surrounded by the right people.
He had become a lightening rod for the left, somewhat similar to Bush but even worse. And just why was he a lightening rod? Because of his CONSERVATIVE agenda & the fact he had kicked their butts so bad.