Ever read Canon XXXIII of the Council of Elvira, 295-302 AD?
No, I haven't. Councils were a dime a dozen in those days, and whatever those people may have opined, mandatory celibacy in the west didn't happen until the counter-reformation. Remember Athanasius?
Sure there have been people talking of it all along, but they did not get their way. We can agree or disagree on the subject, but the attributing abolute authority to a trivial historical factoid doesn't help us reach an understanding. Do I think celibacy has many advantages for those who can maintain it? Sure. Is it worth having the church be run by networks of pooftahs? Hell no, imo.