Then, I guess Jesus made a mistake in building His Church and giving the keys.
Nope, Jesus has never made a single mistake.
We, however, ARE mistaken to think that Peter was infallible in areas of doctrine.