The unanimous consent of the fathers actually opposes the Roman Catholic interpretation of Matthew 16:18 and its attendant ecclesiology. Augustine is typical of the fathers as a whole in this interpretation of Mt. 16:18:
Remember, in this man Peter, the rock. He's the one, you see, who on being questioned by the Lord about who the disciples said he was, replied, 'You are the Christ, the Son of the living God.' On hearing this, Jesus said to him, 'Blessed are you, Simon Bar Jona, because flesh and blood did not reveal it to you, but my Father who is in heaven. And I tell you...You are Peter, Rocky, and on this rock I shall build my Church, and the gates of the underworld will not conquer her. To you shall I give the keys of the kingdom. Whatever you bind on earth shall also be bound in heaven; whatever you loose on earth shall also be loosed in heaven (Mt 16:15-19). In Peter, Rocky, we see our attention drawn to the rock. Now the apostle Paul says about the former people, They drank from the spiritual rock that was following them; but the rock was Christ (1 Cor 10:4). So this disciple is called Rocky from the rock, like Christian from Christ...Why have I wanted to make this little introduction? In order to suggest to you that in Peter the Church is to be recognized. Christ, you see, built his Church not on a man but on Peters confession. What is Peters confession? You are the Christ, the Son of the living God. Theres the rock for you, theres the foundation, theres where the Church has been built, which the gates of the underworld cannot conquer.14
These comments by Augustine are highly significant. He does state that Peter is the rock. But he goes on to explain what he means by that statement. It does not mean that the Church is built on Peters person but on his confession of faith in Jesus Christ. http://www.christiantruth.com/Beckwith-Response-to-Return-to-Rome.html
You may disagree, and I myself do no make CFs determinative of doctrine (nor does Rome actually), but if had have read what he said before you could have more accurately presented his polemic.
Later
His polemic is all "lawyered" up. When Christ named Peter(the Rock)from Simon. He is addressing someone outside of himself because Christ is the builder.
Matthew 18 And I tell you, you are Peter(The Rock),[d] and on this rock[e] I will BUILD MY church, and the powers of death[f] shall not prevail against it.[g] 19 I will give you the keys of the kingdom of heaven,[h] and whatever you bind on earth shall be bound in heaven, and whatever you loose on earth shall be loosed in heaven.
He is addressing "The Rock"that was named by him. Peter.