If they want to remove a priest, with just an allegation, they should do it quietly, working on the most recent allegation, then start checking in other parishes in which he's served. He should not be publicly accused without serious evidence.
Once an allegation is made public, there is no where the priest can go to get his reputation back when the allegation proves false, or there was nothing serious about what might have happened. In many Dioceses, Bishops have turned over their Pastoral responsibility to Commissions who treat priests as guilty before any real evidence is even produced. This is the flip side of the Bishops turning blind eyes to actual abuse in the past. Both are wrong. The Bishop should never abrogate his responsibility to his priests, and should treat each situation personally, with judgement, and discretion, until such time as clear evidence of abuse is produced.
That's true of anyone who is accused of a crime of this nature.
Perhaps people, priests especially, should spend more time making sure their actions are above reproach.