The reason why a marriage can be annulled is because the ministers of the sacrament of marriage are not the priest (who merely stands as the Church's witness) but the two baptized Christians who contract the marriage by their vows.
If one of them never truly intended to enter into a Christian marriage, then there never was a marriage and the sacrament never took place.
In the case of ordination, the minister is the bishop and the ordinand is merely a recipient of the sacrament, not a minister. What the bishop accomplishes in the sacrament is ex opere operato, indelible, and cannot be undone
If the ordinand is not a valid candidate (i.e. mentally disfigured by perversion, in this case) then he is guilty of fraud and the Church is justified in removing the benefits of the sacrament from him, including any rights, dignities and duties accruing to the priesthood, without appeal.