Your observation would be valid in merely human terms. But the issue takes on a different level of understanding of "trust" in God's Providence here, insofar as it treats to matters of "faith." As someone who has rejected Christianity outright, I would imagine that this specific issue of magisterial authority makes no difference to you. Yet you have similar issues regarding "authority" in your own cirlcle. The Old Testament in general, and the Torah in particular, have meaning for you, I would expect. But they do not for Hindus, Shintos, Buddhists, and a host of others. You would have to establish credibility for your Scriptures with such people in the same way a Catholic needs to establish the authority of Sacred Tradition, Ecumenical Councils, the Papacy and the like with non-Catholic Christians.
Catholics consider the sources above to come down to us with Divine sponsorship and ratification. We do not suppose that each individual priest is so endowed. So, like you vis-a-vis rabbinical pronouncements, I don't necessarily agree with every pronouncement issuing from the mouth of every individual priest talking out of his hat, either.
I would equate Catholic traditions such as Peter in Rome, Perpetual Virginity, Immortality of Souls, etc., etc., to Judaism's Jewish "Midrash". Midrash being traditions that embellish Torah and Tanakh. These traditions, some and maybe most might be true, are not binding. The difference it seems is that "all" Catholic tradition has to be accepted to maintain being a Catholic.