I remember as a child how we use to play "telephone". We would line up and the first person would whisper something in the ear of the person next to him. Each succeeding person would do the same thing in turn until it got to the end where the person would repeat the message. By the time the message got to the end, it was so distorted that it no longer resembled the message as it was begun.
This was the basis for closing the books on the scriptures. Augustine stated that the life of Christ was like a pebble cast into a lake. The farther out from the impact of the stone, the more distorted the message becomes. Consequently, the fathers wisely knew that the farther out from the life of Christ, the more distorted our theology would become. It was on this basis they gathered up all the inspired writings and compiled the Bible. (BTW-The same thing happened to the Jews.)
I would content that the early fathers were correct and the only thing you can go back to is the 1st century inspired writings. That is not to say that you can glean certain truths from the fathers throughout the ages. However one can clearly see from history (even in Catholic and Orthodox history) that there have been moments of great inspiration and darkness. But the only way you can discern what these are is through the written word.
HD, the Apostles did write what they knew, so that the truths passed on to them could be read in churches, even while they were still alive, since they couldn't be in all the churches at the same time! The Church also wrote down these truths in its liturgical worship.
The only people who knew what to write "by rote" were the Apostles, and only their work was considered inspired. The problem was not that we would forget or distort that which was already written, but the fact that there appeared forgeries masquerading as "apostolic" that were being read in churches.
The Church had to separate those that were known to be of Apostolic origin and pondered for centuries those writings that appeared Apostolic before fully accepting them or rejecting them. The Gospels were a "no brainer." They were used in churches before the end of the 1st century while the Apostles were still alive and with bishops who were directly ordained by the Apostles.
The problems started in the 2nd century when the Apostles were already gone from this life, and could no longer verify the manuscripts floating around bearing their names as authors. The second secontury saw a proliferation of heresies with a multitude of "apostolic" manuscripts (scrolls) in various churches being read as genuine "epistles" and "gospels."
The Church had no choice but to undertake a formidable task of separating texts that were known to be Apostolic and rejecting those that were not from being read publicly in churches and corrupting the faith.
The problem with that task, and the reason it took three hundred years to complete, were the manuscripts other than the Gospels and Pauline Epistles, such as 1 and 2 Peter and 1, 2, 3 John and the Revelation of John, etc. which did not fully match known works of these individuals, or did not have exactly the same style, language and so on.
There were instances when the churches, even the church in Rome, read manuscripts that were later discarded. So, it is not that it was all verbal and oral and then someone decided to make sure the church "telephone" did not result in distorted message, but the fact that distorted messages appeared in writing purporting to be Apostolic in authorship.
You are correct that the Scriptures were limited to those writings of Apostolic origin and that taught the Apostolic faith. Nothing written outside of the immedicate circle of the eyewitnesses became Scripture.
The recognition and preservation of what was and was not Scripture was the "work" of the Church as a whole. The declarations of any given council of bishops ratified, so to speak, the mystical understanding of the Church -- which is the Mystical Body of Christ. This is perhaps a difference between traditional Orthodox understanding and a slightly different emphasis on the role of formal Church structure held by Catholics. Both are positions Protestants would disagree with, but there is perhaps some difference.
Your analogy of the telephone is not a very exact one, in one sense (but then, neither was mine of the father to son transmission of a family story.)
What happened was more like this: The Apostles received the faith from Christ. No matter where they went throughout the known world, the faith they taught was the same. The key things were written down and compiled into what we now know as Scripture. These Scriptures were shared throughout the Christian world, and were recognized as Scripture by all, eventually. Their authority is paramount and is on a different level from all other parts of Tradition. Nothing can be taught that is not compatible with Scriptural teaching.
But many things were not written down, since the world could not contain the books. Faithful and clergy alike were taught these things throughout the Church as it grew and spread. The deposit of faith, which is Christ Himself, lived within the Church. As controversies arose, the tests to which doctrines were put were first and foremost the writings of the Holy Scriptures, Old and New Testament alike. Were this not so, the Fathers' writings wouldn't be leavened with the words of Scripture. The Church is the humble preserver of the Scriptures and the apostolic understanding of the words of Scripture, but no body of men, regardless of their titles, can place themselves above the Scriptures -- which are the verbal icon of Christ.
Another test was what was believed everywhere and by all, to borrow from the phrase of St. Vicent of Lerins.
This is the "consensus patrum." It was and is not one man on the end of a play telephone chain -- it was what was found to have been passed down throughout the Church. If the same belief had been passed down within the churches in Ephesus, Jerusalem, Rome, Alexandria, Carthage, and the farthest reaches of the Church, it was and is pretty safe to say that the only possible explanation for this is the explanation that the Church gives: that these beliefs are Apostolic in origin, and were thus passed down as a precious deposit.
Given the tendency of the Church in the early centuries to hash things out in great detail over seemingly minor issues, it should be notable that there is no record of internal debate or polemics about any of these things. When one branch of the "telephone chain" got something wrong, the other branches were usually right there to set them straight.
And more to the point, since Christ himself is, in a very real sense, Himself the deposit of the faith, and lives within the Church through his Holy Spirit, these beliefs and teachings are not so much things that are passed down as they are things that remain living within the Church.
For someone who does not live within the liturgical and ascetic life of the Orthodox Church, it is perhaps hard to understand how alive and present Holy Tradition (including the Scriptures themselves) is to us. It is not the province of musty manuscripts and stories told over the breaking of bread (although there have always been plenty of both of those!) It is primarily the province of the living experience of the Church at prayer and union with God.