That doesn't work, because then you would have to explain why Christ said it was St. Peter: "Tu es Petrus. . ." and so on. Our Lord could have said "I am the Rock" instead. But He didn't.
The Church is built on Christ, not Peter.
The Church is ultimately built on Christ and His grace, but it is physically to be built, as He explains, on Peter. That is analogous to when He said to the Apostles in the Upper Room after the Resurrection in John 20:21-23:
"As the Father hath sent me, I also send you. When he had said this, he breathed on them; and he said to them: Receive ye the Holy Ghost. Whose sins you shall forgive, they are forgiven them; and whose sins you shall retain, they are retained."
As Christians, we are all His (imperfect) agents, whether we are laymen reading the Bible or successors of St. Peter hearing confessions. It's our job to cooperate with His grace, offering our lives in return. It's hard work, but the payoff is, as you know, out of this world.
>>The Church is ultimately built on Christ and His grace, but it is physically to be built, as He explains, on Peter. <<
I could agree with that statement within the context of “all Christians are, by definition, saints.” The “church” is the body of believers.
As the sign over the inside door jam of one church I attended says (as one walks out the door), “The Church has left the building!”
“Thou art Peter (Petros) and upon this rock (Petra)I will build my church...”
Christ used 2 different words. You can look it up.