Where is it stated that this is only to Peter?
Consider Matthew 18:18-20
“Assuredly, I say to you, whatever you bind on earth will be bound in heaven, and whatever you loose on earth will be loosed in heaven. “Again I say to you that if two of you agree on earth concerning anything that they ask, it will be done for them by My Father in heaven. For where two or three are gathered together in My name, I am there in the midst of them.”
That doesn't sound exclusive to Peter. And you bind and loose using keys...
But what about the renaming episode?
Peter is the first who showed faith saying Jesus is God, and is granted the name as an echo of Jesus (the rock) as we are all in calling ourselves Christians (little Christs). The foundation is Christ, and the Church is built though belief as Peter showed in Matt. 16.
Matthew 18:18f is not exclusive to Peter. The issue was the scriptural authority given the Church. Here it is.