Just wanted to say that you can't be surprised others would not trust the Church's contention that Peter was in Rome? I would not blindly accept everything from Church fathers. I'd rather take a pick and choose approach. After all why would one accept everything from the same people that passed down immortality of souls or perpetual virginity to cite a couple of wacky examples. FYI I don't accept every rabbinical tradition passed down to me either. There's trememdous freedom in this approach.
Your observation would be valid in merely human terms. But the issue takes on a different level of understanding of "trust" in God's Providence here, insofar as it treats to matters of "faith." As someone who has rejected Christianity outright, I would imagine that this specific issue of magisterial authority makes no difference to you. Yet you have similar issues regarding "authority" in your own cirlcle. The Old Testament in general, and the Torah in particular, have meaning for you, I would expect. But they do not for Hindus, Shintos, Buddhists, and a host of others. You would have to establish credibility for your Scriptures with such people in the same way a Catholic needs to establish the authority of Sacred Tradition, Ecumenical Councils, the Papacy and the like with non-Catholic Christians.
Catholics consider the sources above to come down to us with Divine sponsorship and ratification. We do not suppose that each individual priest is so endowed. So, like you vis-a-vis rabbinical pronouncements, I don't necessarily agree with every pronouncement issuing from the mouth of every individual priest talking out of his hat, either.