He gave the power to bind and loosen ("the keys") to the original set of Apostles (minus Judas). "The keys" are not physical keys but the authority from God. The Apostles, in turn, passed that on to their successors, by the same authority.
He gave them to Peter, along with the charge to feed His sheep. Your position on it, that they were given to all of the Apostles & their successors is an interesting innovation.