So you are saying that Matt 16:19 "And I will give you the keys of the kingdom of heaven, and whatever you bind on earth will be bound in heaven, and whatever you loose on earth will be loosed[d] in heaven. " Is the entire justification for everything that is the office of pope? Well I shouldn't be surprised considering how much Marianism has been built based on "hail Mary full of grace, blessed are you among women".
It seems to me that by merely being the first to make the wild claim about the pope verse, one gets to own it and all the doctrine they can contrive about the pope.
No. There is a good deal more inferential Scriptural evidence, as well as historical evidence from the very earliest days of the Church, but that verse is sufficient in itself.
I've seen MANY Protestant positions justifed based on a lot flimsier scriptural evidence.
Two good books:
"Upon This Rock"- Steve Ray
"Jesus, Peter and the Keys" by Scott Butler, Norman Dahlgren, and David Hess
Both of which have scores of references to scripture, history, and linguistics. The evidence is incontrovertible.