If Jesus was going to give authority to men, he had no other kind to choose from.
The passage is clearly a grant of authority to Peter. It doesn't say "Your confession of faith is the key to the kingdom of heaven, but "*I* will give to *you* the keys of the kingdom of heaven". Any reasonable reading of the text sees that.
Nowhere does Scripture suggest that God chose a sin-prone and fallible foundation upon which to build His church.