Responsibility is the key word. All are responsible for the actions of those chosen to represent us. If you choose not to vote for a candidate who will staunchly defend your rights, you have earned no immunity to the repercussions of his actions.
The recall process is always open to the citizens. In some jurisdictions it is well defined and often used. Should you have the misfortune of living in a place where there is no well defined process, you may as a last resort join with others and form your own vigilance committee.
What if the top 30% of productive citizens don't want a crook in office, and the bottom 70% (many of whom produce nothing) do? Should the top 30% have to pay for the actions of the other 70% (whose actions may well be predicated upon the fact that, for the most part, they don't have to pay for them?)