Let me try a different analogy to make my point. Isn't a celibate saying he's a homosexual a bit like a tee-totaller saying he's an alcoholic? Let's say he's never had a drink before in his life - he just somehow knows he's an alcoholic.
Uh, no. A homosexual is attracted to males, but it is possible for a homosexual to have never acted on that attraction.
Heterosexuals are attracted to women, but it is possible for a heterosexual to have never acted on that attraction.
If a man never takes a drink, how will he know if he would be so attracted to drink that he would become addicted to it?
If a non-drinker got an erection every time he spotted an alcoholic drink, he would know that he had a problem.