Speakers of the House do not normally have fixed terms. Usually, when deposed from the job, the immediate resignation follows (not so with Nancy Pelosi, of course, but then that lady is exceptional in a number of ways, none of which I count as good), and it expires anyway if not re-elected from the home district.
There is, however, something like a vote of “no confidence”, a parliamentary move that essentially strips the Speaker of his (or her) post. Something like that was about to be pursued against John Boehner, and would have been done against the Democrat Speaker Jim Wright in 1989, when each were burned on ethics charges and an internal party dispute.
Thanks for the answer to my questions.