Incorrect. Paul's instruction to Timothy and Titus was not an absolute. What it was was an instruction not to pick men who were divorced or remarried widowers. Refer to Canons XXVII and XXXIII of the Council of Elvira, 295-302 AD. Also, from the Council of Carthage, 390 AD, regarding celibacy and the discipline of lex continentiae: "ut quod apostoli docuerunt, et ipsa servavit antiquitas nos quoque custodiamus". Which translated reads "What the apostles taught and what antiquity itself observed, let us also continue."
Suggest you read Cochini's "The Apostolic Origins of Preistly Celibacy" for some much needed edification.
It was also a guard against polygamy which had not been fully eradicated in some pagan and Jewish circles.