Could the answer to your question be:
"The meaning of the passage is not that man can forgive sins that belongs only to God (Isaiah 43:23) but that they should be inspired; that in founding the church (meaning church universal, not just the RC church), and in declaring the will of God, they should be taught by the Holy Spirit to declare on what terms, to what characters, and to what temper of mind God would extend forgiveness of sins. It was not authority to forgive individuals, but to establish in all the churches the terms and conditions on which men might be pardoned, with a promise that God would confirm all that they taught; that all might have assurance of forgiveness who would comply with those terms; and that those who did not comply should not be forgiven, but that their sins should be retained.
This commission is as far as possible from the authority which the Roman Catholic claims of remitting sin and of pronouncing pardon."
That quote frames the question nicely. The Church would agree with much of it, I think, but might clarify that the Holy Spirit’s gifts flow through other people to all of us, and that authorized men are the ones who we can rely on to channel the forgiveness gift. Centuries of scholarly work and inspired practice is used to back this up.