The Bible says that upon the confession that Jesus is Lord, Christ will build His church.
Not upon Peter. Peter is but a man.
Peter was not impeccable, he made human mistakes.
But as Pope when acting on faith and morals, he was guided by the Holy Spirit and thus was in falliable, as is the Magisterium and Pope of today when ruling on major items of faith and morals. Some people don’t realize that.