What Roman Catholic group did Christ say that to? The answer is none, for He was speaking directly to Peter.
If by that you mean to limit that to when Christ was speaking to individuals, prior to the Crucifixion and Ascension, then the same could be said for anyone, at any time since His Ascension, returning then to where He was before.
In John 6:62 Jesus is attributed to asking of His disciples;
Yes, what then.
Except for when they were not, but instead had rubber-stamped whatever it is that they decided to have been said to come from God, or was otherwise claimed to have been established by God from the onset --- like --- the papacy itself, which is much (theologically) refuted by the contents of Matthew 18.
The earliest Church understood things differently then 'Rome' alone came to view themselves. From time of the last Apostle's demise shall we say, Church polity was horizontally conciliar for many long centuries.
The first efforts by Rome to assert it's own prerogative and singular authority over others, were widely rebuffed.
I should have written;
instead of "then"...Rome alone...
Matthew 16:18 (Spoken to Peter) "I will give you the keys of the kingdom of heaven; and whatever you bind on earth shall have been bound in heaven, and whatever you loose on earth shall have been loosed in heaven."
Matthew 18:18 (The verse I actually quoted spoken to ALL THE APOSTLES) "Truly I say to you, whatever you bind on earth shall have been bound in heaven; and whatever you loose on earth shall have been loosed in heaven.
Further Luke 9:1-2 Then He called His twelve disciples together and gave them power and authority over all demons, and to cure diseases. He sent them to preach the kingdom of God and to heal the sick.
These 12 disciples were the DIRECT predecessors of the today's Bishops and Cardinals. Not a single protestant group can trace a direct line back to the apostles Only the Catholic Church can do this.