It’s been done.
Tom Foley comes to mind.
Rostenkowski was another speaker that got tossed.
But GOP voters can join with Democrats (if they have a candidate) and vote Ryan out.