In the Bible, Samson was nazir -- that is a person who lives a sort-of "holier" than normal life. A nazir swears off alcohol, doen't cut his hair, etc. But a nazir, is required to make a sin-offering, that is -- even the partial denial of some of life's allowed pleasures is a sin -- a missed opportunity. The inablility to enjoy wine moderately is a failing -- whether by abstinence or by drunkeness.
Life is lived for the refinement of the soul, but by fully living one's life one can more fully refine one's soul.
So yes, groups that teach extreme complete avoidance of some things like alcohol, sex (the Shakers), etc. rather than teaching ways to acheive moderate or appropriate use -- those groups will have problems.