In John 20, as we shall see.
First, what are the "keys to the kingdom"? They are defined for us in Matthew 16:19
"And I will give you the keys of the kingdom of heaven, and whatever you bind on earth will be bound in heaven, and whatever you loose on earth will be loosed in heaven."
Clearly, the keys to the kingdom are the ability to bind or loose in heaven.
So when we read in John 20:19-23
Then, the same day at evening, being the first day of the week, when the doors were shut where the disciples were assembled,for fear of the Jews, Jesus came and stood in the midst, and said to them, "Peace be with you." When He had said this, He showed them His hands and His side. Then the disciples were glad when they saw the Lord. So Jesus said to them again, "Peace to you! As the Father has sent Me, I also send you." And when He had said this, He breathed on them, and said to them, "Receive the Holy Spirit. If you forgive the sins of any, they are forgiven them; if you retain the sins of any, they are retained."
We see that this has been received by all of the Apostles. They all have the ability bind or loose in heaven. They all have the keys to the kingdom.
And where does any Church Father say anything to back up this startling assertion?
What do you think first among equals meant?
Again, show me a Church Father who said the Pope is "First among equals".
The Keys are a symbol of ruling authority. Thus, being granted the Key to the City. The power to bind and loose is not a power to rule, but to forgive sins, as you correctly state. The Keys relate to the Founding of the Church upon Peter.