The down side is being mis-characterized by a bunch of zealots and being fooled into giving up one of the great pleasures in life, when in fact you had no problem in the first place.
If people notice a significant behavioral change in you, and care enough to point it out, alcohol has ceased being a “Great pleasure of life,” and has turned into a problem. However, if you’re not an alcoholic, there’s no way you can understand this.