Peter was a fallible human being, prone to sin. His confession was of a different nature: divinely inspired:
Matthew 16:17
And Jesus said to him, “Blessed are you, Simon Barjona, because flesh and blood did not reveal this to you, but My Father who is in heaven.
If Jesus was going to give authority to men, he had no other kind to choose from.
The passage is clearly a grant of authority to Peter. It doesn't say "Your confession of faith is the key to the kingdom of heaven, but "*I* will give to *you* the keys of the kingdom of heaven". Any reasonable reading of the text sees that.