I see your point, but would argue that for most of the history of the church many clergy were cynical, corrupt, or had other terrible failings. When did the Jesuit Order ever have a good reputation?
I suppose it is not limited to just the church and clergy. In several empires, eunuchs were renowned for their service, but always with the warning that they should never be given real power.
So is bad the price of good? Perhaps. But to lose good out of the rejection of bad or evil would be a tragedy.
I'm merely arguing to end the celibacy rule for nearly all future ordinations. It's just another HR issue, really. This would certainly address the immediate problem. Like I said above, we can keep celibacy for monks and cloistered nuns, but the price of that is that they not have regular contact with laity.