Matthew 20:25-26: But Jesus called them to Himself and said, "You know that the rulers of the Gentiles lord it over them, and those who are great exercise authority over them. Yet it shall not be so among you; but whoever desires to become great among you, let him be your servant."
So our Lord tells them that those who act as rulers should do so as servants of those they rule. Thus the Peter and the popes, when exercising their office, should truly act as the Servant of the Servants of God. Or are you saying that Christians should never hold any offices of authority? No pope, nor patriarchs, nor bishops, nor kings, nor presidents, etc.?
Also note well that when this story is recounted by Mark (11:35-45) our Lord does not deny that someone will sit at his right and his left but that "to sit at my right hand or at my left is not mine to grant, but it is for those for whom it has been prepared."
Nor should we forget the keys given to Peter in Matthew 16 recall the key given to Eliakim in Isaiah 22 as the symbol of his authority as master of the palace.