I’d look at it like this.
James was the Bishop of Jerusalem, just like Paul was the Bishop of the Gentiles. Each Apostle had his own territory. Origen of Alexandria (ca. 185-254) writes that the apostles divided up the work of evangelizing the world between them — Peter, for example, took Pontus, Galatia, Cappadocia, and Bithynia, and at “the last came to Rome, and was crucified head-downwards; for he requested that he might suffer thus.” According to Origen, other apostles went elsewhere; Thomas was assigned Parthia (today’s India), and John was given “Asia.”
The Pope is still a Bishop of the Church - the Servant of the Servants of God. He is not a tyrant in the secular sense, he is a leader - we believe chosen by God, but still a leader, not a dictator.
Peter was the leader of the Apostles (see proofs in New Advent and elsewhere if required) and is mentioned in the Gospels many more times than all the other Apostles combined. His leadership, confirmed in various ways by Jesus, simply carried on after the Ascension.
His leadership followed him, obviously, and as he was Bishop of Rome when he died, his successor, as Bishop of Rome assumed the mantle of leadership as well. Now, he is first amongst equals with the other Patriarchs of the Churches - a very real thing now, and getting closer to formal declaration.