Christ gave it directly to Peter. Please read that section again.
Christ did give the mantle of leadership to Peter. That much is not in question. Peter took over as head of the Church. That’s the way Jesus set it up.
However, Peter is not the rock Christ is referring to. Jesus is the firm, never moving rock. Not some fallible man.
The real problem with Catholicism isn’t Peter, it’s Linus I.
After Peter died Linus claimed to be the head of the Church. Only problem is, there was still a living Apostle on the earth during this time; John. John was next in line and was the true leader of the Church not Linus.
Linus didn’t continue Christ’s Church, he created his own in the eyes of God.
Even while Linus was claiming to lead the church, Jesus appeared to John and spoke to him face to face. Jesus never appeared to Linus ever. And that’s a fact.
Jesus’ appearance to John proves John’s authority in the eyes of God.