If not, read up about him. He's a fascinating character. His early life was unencumbered by God and grace, his early ministry was popish, and his mature works are Predestinarian and Scriptural.
"We must hold to the Christian religion and to communication in her Church which is Catholic, and which is called Catholic not only by her members but even by all her enemies. For when heretics or the adherents of schisms talk about her, not among themselves but with strangers, willy-nilly they call her nothing else but Catholic. For they would not be understood unless they distinguish her by this name which the whole world employs in her regard." The True Religion, 7,12, 397 A.D.
"This Church is Holy, the One Church, the True Church, the Catholic Church, fighting as she does against all heresies. She can fight, but she cannot be beaten. All heresies are expelled from her, like the useless loppings pruned from a vine. She remains fixed in her root, in her vine, in her love. The gates of hell shall not conquer her." Sermon to Catechumens, on the Creed, 6,14, 395 A.D.
"But in regard to those observances which we carefully attend and which the whole world keeps, and which derive not from Scripture but from tradition, we are given to understand that they are recommended and ordained to be kept either by the Apostles themselves or by plenary Councils, the authority of which is quite vital to the Church." Letter to Januarius 54,1,1, 400 A.D.
"I believe that this practice comes from apostolic tradition, just as so many other practices not found in their writings nor in the councils of their successors, but which, because they are kept by the whole Church everywhere, are believed to have been commended and handed down by the Apostles themselves." Baptism 1,12,20, 400 A.D.
"Before His suffering the Lord Jesus Christ, as you know, chose His disciples, whom He called Apostles. Among these Apostles almost everywhere Peter alone merited to represent the whole Church. For the sake of his representing the whole Church, which he alone could do, he merited to hear, I will give you the keys of the kingdom of Heaven (Matt 16:19)." Sermons 295,2, 391 A.D.
"Since by Christ's favor we are Catholic Christians:" Letter to Vitalis, 217,5,16, 427 A.D.
"By the same word, by the same Sacrament you were born, but you will not come to the same inheritance of eternal life, unless you return to the Catholic Church." Sermons, 3, 391 A.D.
"Tell us straight out that you do not believe in the Gospel of Christ; for you believe what you want in the Gospel and disbelieve what you want. You believe in yourself rather than in the Gospel." Against Faustus, 17, 3, 400 A.D.
"Adam sleeps that Eve may be formed; Christ dies that the Church may be formed. Eve is formed from the side of the sleeping Adam; the side of the dead Christ is pierced by the lance, so that the Sacraments may flow out, of which the Church is formed." Homilies on the Gospel of John, 9,10, 416 A.D.
"What the soul is to man's body, the Holy Spirit is to the Body of Christ, which is the Church. The Holy Spirit does in the whole Church what the soul does in all members of one body. But see what you must beware of, see what you must take note of, see what you must fear. It happens that in the human body, or rather, off the body, some member, whether hand, finger, or foot, may be cut away. And if a member be cut off, does the soul go with it? When the member was in the body, it lived; and off, its life is lost. So too, a Christian man is Catholic while he lives in the body; cut off, he is made a heretic; the Spirit does not follow an amputated member." Sermons, 267, 4, 391-430 A.D.
"Let us not listen to those who deny that the Church of GOD is able to forgive all sins. They are wretched indeed, because they do not recognize in Peter the rock and they refuse to believe that the keys of the kingdom of heaven, lost from their own hands, have been given to the Church." Christian Combat 31,33, 396 A.D.
"The Catholic Church is the work of Divine Providence, achieved through the prophecies of the prophets, through the Incarnation and the teaching of Christ, through the journeys of the Apostles, through the suffering, the crosses, the blood and the death of the martyrs, through the admirable lives of the saints. When, then, we see so much help on God's part, so much progress and so much fruit, shall we hesitate to bury ourselves in the bosom of that Church? For starting from the Apostolic Chair down through successions of bishops, even unto the open confession of all mankind, it has possessed the crown of teaching authority." The Advantage of Believing, 391 A.D.
It is difficult to read Augustine and even consider trying to be a non Catholic Christian...