You are using an argument that most would not agree is true to support your claim, namely that the popes claim a direct line of succession from the apostles. And that is a key difference for Catholics and non-Catholics. And keep in mind that the doctrines concerning Mary came about almost 300 years after Christ.
Catholic doctrines have not typically been formally defined on day 1. They are usually formally defined after a heretic rejects what the church already believes.