Physical attraction plays a part for both men and women, regardless of whether they are celibate or promiscuous. Whether or not it is less of a factor for celibates, I claim no real knowledge. My guess would be that it varies with the individual, the same as what is "attractive" varies from indivudual to individual. We've all seen couples that we can't help but wonder what each sees in the other, but that's because we have a different idea of what's attractive (to us) than they do.
I still say that if being celibate makes physical attraction more of a factor, then, in the case of men, you are actually reinfocing the idea that men are slaves to their sexual appetites, at least to some degree. I don't think that's necessarily true, nor should it be. Do I have a sexual appetite? Certainly! I was married for 16 years, and fathered 3 children. We all know how that happened. I've been divorced for 12 years. Have I had sex in those 12 years? Yes, and I've learned through painful reality that it isn't in my best interest to do so outside of marriage. That's why I support the idea of celibacy. Is it difficult to abstain? Yes! But I feel better about myself, and if things don't work out between me and someone I date, it doesn't hurt near as much, because there's no spiritual union to rip apart. Not only that, but I've found that the woman isn't hurt as much either, and several of those women remain good friends.
It's all about boundaries. Setting boundaries makes it easier to truly get to know someone, because having sex by the third date only confuses the issue. I don't buy this idea that people have to "find out if they're sexually compatible". What's to find out? When a man and a woman get married, they learn how to please their partner, what they like, etc., and are free to explore it, safe in their love for each other. The equipment is basically the same, with only minor variations.
your honesty and insights add much to the discussion.