If yoiu think that's an offense then heterosexual priests with desires would not be suitable either.
I don't believe you can separate homosexual orientation from the behavior. Heterosexuality is not, in an of itself, sinful. Homosexuality is sinful, against nature and against God's law. And those who have an orientation toward sin should be held accountable. In priests, having an orientation towards heterosexuality is not wrong, but normal. Repressing that sexuality for the purpose of serving God and taking vows to only serve Him is a sacrifice and solemn commitment.
By the way, I don't believe homosexuality is an inbred gene or that a man is "born that way." But I'll leave that topic for other discussions which are plentiful around FR.