Not the same office.
So what? He was elected every time. Maybe that's because he was a good, competent, and valuable man. Yet you would toss him because you don't trust your fellow voters.
Maybe the competence of those voters is the real problem, one that term limits can make worse (see "California").
It's not the fix you hope it would be. While I understand the sentiment, our energies would be more effective logically redirected.