Jesus said this with all 12 apostles being present, it is a metaphor to tell them that they were in control of their heavenly destiny not the Temple. It is now PERSONAL! not Roman Catholic, or Peter's.
If you seek you will find, it starts understanding the Jewishness of our faith and the context of the message.
Really? Then why didn't Jesus tell everyone they had the keys to the kingdom of heaven instead of speaking directly to Peter (who is the only one, ever, in the Bible who receives the "keys" to the kingdom of heaven). Furthermore, if you look at the context, Jesus had just changed Simon's name to "rock," as a sure foundation on which to build His kingdom. He does not say this of any other apostle/disciple ever. Combine this with the "keys" statement that follows immediately afterward and logic would dictate that Jesus was speaking solely to Peter.
Sure noob.