The sex drive like many other appetites (with the exception of eating) loses its pull on a person when they don't engage in it for a long period of time. Most people require affection for sex to be satisfying; one night stands or casual sex do not provide much of this sought after component; especially in people over the age of 25 or 30. Having sex for the sake of the act itself is no different than eating when you're not really hungry. Too much of that and you become fat; too much sex without meaning and you become hollowed out. Celibacy is not that hard in my opinion; once the intital period of adjustment has passed.
Speak for yourself.