Jesus said that the Church would be key in the building of the Church. That happened. You can read about it in the book of Acts.
At no place I can remember does Jesus say that anyone who follows Peter succeeds Peter in that role of getting the Church off the ground.
I am open to any scripture you might have that demonstrates Peter is to be succeeded by others with that same role and position.
Apostolic Succession has been the docrine of the Church from the first century. The Traditions of the Church are the enacted consensus of the Apostles, Disciples and the Early Church Christians and Fathers. Doctrines are never "settled" until they are seriously challenged.
Consider that in 97 AD serious dissensions arose within the Church of Corinth and although the Apostle St. John was still living in Ephesus they appealed to the Roman Bishop, St. Clement, who wrote an authoritative letter to restore peace. St. John did not interfere with St. Clement or with Corinth.
Further, before 117 AD, St. Ignatius of Antioch addressed the Roman Church as the one which "presides over charity . . . which has never deceived anyone, which has taught others."
Additionally, St. Irenaeus in about 190 AD wrote the theory and practice of doctrinal unity as follows:
"With this Church [of Rome] because of its more powerful principality, every Church must agree, that is the faithful everywhere, in this [i. e. in communion with the Roman Church] the tradition of the Apostles has ever been preserved by those on every side." (Adv. Haereses, III)
Peace be with you