If that's the case, your bone to pick is with Christ Himself for "foolishly" giving Peter the, um, KEYS to the kingdom of Heaven. If it's an either/or with you, then you need to hang your hat on the words of Paul or the words of Christ, take your pick. If you believe that Scripture cannot contradict itself, then there has to be a relationship between Paul's words and Christ's words which demonstrate correctness in both statements. That's the problem with relying on a black-and-white approach to the Gospel. You read it like it came off an IBM supercomputer, and not imbued with a specific intent of a human author inspired by the Holy Spirit.
If that's the case, your bone to pick is with Christ Himself for "foolishly" giving Peter the, um, KEYS to the kingdom of Heaven.
But Christ did not give the "keys" exclusively to Peter, and the early church fathers, you know, "tradition" did not believe that later assertion by Rome either, as I have already cited.
So much for the "unanimous consent of the fathers".