The council of Elvira went further:
Already in 305 A.D. (thats very early), before the Churchs liberation under Constantine, the Council of Elvira in Spain passed the following decree: That bishops, priests and deacons, and in general all the clergy, who are specially employed in the service of the altar, abstain from conjugal intercourse. Let those who persist be degraded from the ranks of the clergy (Can. 33). And by the end of the fourth century, the Second Council of Carthage in Africa declared, What the apostles taught in the early Church preserved, let us too observe.
For further background on celibacy check out: Link to Fr. Hardon archive on celibacy