See here ----> http://www.aomin.org/Epitetaute.html
Let's be crystal clear here. No Greek scholar in the entire world claims that the pronouns and verbs in verses 17-19 are not singular. No one! Not even James White. Only someone who had no clue about Greek would even say that any of the 2nd person pronouns in verses 17-19 are plural. No Greek scholar claims that in verses 17-19, Jesus is talking to anyone other than Peter.
The point of disagreement is not at all whether in vs. 17-19 Jesus is talking only to Peter. James White (and some other Protestant Greek scholars, though not all) disagree with with Catholics regarding the referent of [epi taute te petra] "upon this rock". Catholic scholars say that the referent of 'taute te petra' is Peter. Some Protestant scholars say that the referent of 'taute te petra' is either Peter's profession of Christ's identity or Peter's faith.
-A8
Touche' as to which person is indicated in the passage. For now, I'll take your word on that. However, you make my point that there are Greek scholars on both sides of the issue as to meaning of the passage.