Will Durant is very far from balanced in his view of the Church. The real origin of priestly celibacy is far more complex and nuanced (and *older*) than he admits. As a counterbalance, I'm told that a good book is Christian Conchini, SJ's Apostolic Origins of Priestly Celibacy. (I haven't read it.)
What you say is true, that celibacy is a discipline that could be ended by papal edict. However, there are many good reasons for keeping it.
Being a Protestant it is a moot point to me, but I've got to say that celibacy seems to me to be unnatural, priest or layman. If I've offended you, I'm truly sorry.