Fair enough. But then you'd get into a situation of legislators running for office on the platform of 'elect me and I'll go after Judge so-and-so'. It seems like this is a 'Merchant of Venice' situation where the law may be on the side of unpleasant consequences.
You'd need a majority, and given the lag time between elections, it would be extraordinarily difficult to sustain one in favor of impeaching a judge, without sufficient cause.
Judges have to be answerable to someone, sometime, or they will - as they have - decide to make law instead of performing their constitutionally ordained role of interpreting it.