Jesus ultimately holds the keys. He is "the power behind the keys," just as the king of the House of David in the Old Testament was "the power behind the keys" which were given to his vice-regent. This is clear in Isaiah 22:
Isaiah 22 : New International Version (NIV)Note also that he will be a "seat of honor for the house of his father" (like the seat of Moses* and chair of Peter) and that he will "be like a father" (father or "papa" or pope) to the house of Judah.A Prophecy About Jerusalem
15 This is what the Lord, the LORD Almighty, says:
"Go, say to this steward, to Shebna, who is in charge of the palace:
16 What are you doing here and who gave you permission to cut out a grave for yourself here, hewing your grave on the height and chiseling your resting place in the rock?
17 "Beware, the LORD is about to take firm hold of you and hurl you away, O you mighty man.
18 He will roll you up tightly like a ball and throw you into a large country. There you will die and there your splendid chariots will remain- you disgrace to your master's house!
19 I will depose you from your office, and you will be ousted from your position.
20 "In that day I will summon my servant, Eliakim son of Hilkiah.
21 I will clothe him with your robe and fasten your sash around him and hand your authority over to him. He will be a father to those who live in Jerusalem and to the house of Judah.
22 I will place on his shoulder the key to the house of David; what he opens no one can shut, and what he shuts no one can open.
23 I will drive him like a peg into a firm place; he will be a seat of honor for the house of his father.
24 All the glory of his family will hang on him: its offspring and offshoots-all its lesser vessels, from the bowls to all the jars.
25 "In that day," declares the LORD Almighty, "the peg driven into the firm place will give way; it will be sheared off and will fall, and the load hanging on it will be cut down." The LORD has spoken.
_______________________________________
*Matthew 23:1 Then Jesus said to the crowds and to his disciples: 2"The teachers of the law and the Pharisees sit in Moses' seat. 3 So you must obey them and do everything they tell you.
BTW I think the keys were representive of teaching authority..(the keys to the room where the scrolls were kept were a sign of authority in the temple I believe..) So that authority was meant for the building of the church and I see no scripture to indicate that Peter had any permission to pass them on
Isaiah 22:19 above indicates that the possessor of the keys held the office of vice-regent. An office implies succession and, in fact, that is what we see here. Shebna's office is given to Eliakim.
The sin and excesses and false teaching that led to the reformation may well have moved that authority..
"And, since it is he that commits the government into their hand, they must administer it according to his will, for his glory; they must judge for him by whom they judge and decree justice, Prov. 8:15. And they may depend upon him to furnish them for what he calls them to, according to this promise: I will clothe him; and then it follows, I will strengthen him. Those that are called to places of trust and power should seek unto God for grace to enable them to do the duty of their places; for that ought to be their chief care. "
Matthew Henry
I would submit those keys were withdrawn. The Catholic church was no longer the sole voice of Chrisitanity..the voice was moved to the reformers.