Ryan thinks he’ll somehow inherit the Presidency, eventually, by being Speaker of the House. He’s avoiding doing anything that might hurt his chances down the line. He needs removed.
He'd have to get past Pence first. And I don't see that happening.