It is more than an annoyance. Sharing an office with a smoker irritated my throat, nose and lungs leading to coughing, sneezing, etc. My health WAS affected.
To the larger portion of the general public it is no more than an annoyance.
If you want to nitpick, which is what you have been doing with almost everyone that has replied to you, we can do that.
Again, should we legislate an annoyance because a small minority is affected?