Mandatory celibacy has for clergy has no root in the Apostolic Tradition. The Apostle Paul himself, who wished that all men were as he was (1 Corinthians 7:7) namely celibate, also assumes in 1 Timothy 3 that a Bishop would be married. From the very beginning of the Church it was assumed that marriage itself was no impediment to ordination although the Church did, in fact, address the issue of who a Priest could marry (no actresses, etc.) and how many times (basically once) in its various canons on the subject. This is the Tradition of the Eastern Orthodox Churches to this day with the exception that we have canonically nullified married men from the episcopacy for practical but not scriptural or historc reasons.
But is that the central problem of the Catholic Church? I don't think so. Rather the Church has for too long allowed wolves in sheep's clothing into the fold and they have done much to destroy the faith of many. There are any number of clergy and monastics in the Church who have maintained celibacy but not the Faith.
Since St. Paul's teaching on celibacy had not been around long enough to create an entire generation of Christians raised to respect celibacy, it stands to reason that bishops would be drawn from sober adults who had already come of age before they knew Jesus.
There was no existing pool of celibates to draw on for the episcopal office at that time.
The present Eastern Orthodox discipline does not go back to apostolic times but represents a relaxtion of more rigorous mandatory continence within marriage or celibacy that does go back to apostolic practice and teaching.
The assumption that bishops and priests should not be married in order to devote themselves sacrificially to the service of Christ rests on Mt. 19, 1 Cor 7 and the passage in the epistle to Timothy in which St. Paul says a bishop should be the husband of one wife (that is, should not remarry after being widowed, which was a counter-cultural requirement in Graeco-Roman culture but demonstrated self-control and a desire to devote oneself to God, as was also true of women who chose not to remarried after becoming widows and were thereafter supported by the Church on the official "rolls" because their prayers and service to the poor etc. was made possible by their choice not to remarry).
The first legislation requiring celibacy appeared as late as the fourth century (300s), but this is the first legislation that has survived. Until this time (ca. 314) the Church was persecuted, councils had difficulty meeting, and their legislation has been lost. Moreover the legislation of the early 300s describes clerical sexual abstinence, or continence, as a long-established practice, not something first being mandated at this point.
Even married priests were expected, according to this legislation, to abstain from marital relations with their wives. The woman thus exercised a veto power over her husband's ordination. The early legislation admonishes priests who promised continence but were not practicing it, to keep their pledge.
In other words, abstinence from sexual relations for both married and unmarried priests was well established practice long before the first surviving legislation in the early 300s. For an evaluation of every shred of surviving evidence back to apostolic times, see Christian Cochini S.J., Apostolic Origins of Priestly Celibacy [San Francisco: Ignatius Press, 1990] as well as the more recent work by Stefan Heid [1997]).
The same rules about mandatory sexual abstinence for priests were shared by both East and West until the East modified it slightly in the 690s. Even that modification was restricted in scope: priests were still required, in the East to abstain from marital relations on the days on which they “handled sacred things.”
The claim that one of the fathers of the Council of Nicea, a monk-bishop named Paphnutius, favored married and sexually active priests is based on a pious legend, according to the detective work of Cochini and others. For centuries this has been taken as giving great antiquity to the Eastern position. In fact, the modified policy at the Synod of Trullo (not an ecumenical council) in the 690s was an innovation, which is why the bishop of Rome rejected it.
Actually, the cite is "married only once."
This has implications. While your thesis is perfectly legitimate, the cite could ALSO include men who were married "only once" but who are widowers.
Further, there is compelling, but NOT conclusive, evidence, that the ancient rule was "celibate OR continent," meaning that the married ordinand agreed (and wife agreed, too) NOT to enjoy marital relations after Ordination.