Did the Tea Party Gang of 70 go to him and threaten to join the Dem Caucus and depose him as Speaker? That’s the only thing that would work IMO.
If Republicans and Democrats made a deal to vote the most conservative Democrat to be speaker, he would almost certainly be more conservative than Boehner.