You almost got it right, technically that is. The reality is the Speaker has immense power over the entire house and almost total power over his party. Committee assignments are totally in his control for reward or punishment. Few congers critters are willing to put their positions in jeopardy by challenging the incumbent Speaker. Recall that dem James Traficant was stripped of all his committee assignments for voting with the GOP, thus isolating him and making him powerless as well as useless to his constituency back home. The weak and powerless don’t survive.
Aside from Newt can you name one challenger that succeeded? Never happened with the Dems.
An account of GOP Speaker Thomas B. Reed’s tenure would be an interesting and informative read about how much power the Speaker really has.
Best to knock ‘em off in the Nov polls once you make the decision they have to go. Never leave them merely wounded. They have to be stripped of all power lest they cause back bench mischief.
Nits make lice.