The antis tried to get OSHA involved and OSHA told them that if they got involved that OSHA would set PELs (Permissable Exposure Levels) for ETS. The antis backed out of it so fast that the air friction caused hurricane Bertha.
That right there should tell you that it isn't about harm, it's about, "I don't like the smell."
OSHA did, they are right here:
http://www.nycclash.com/smoke_chart.html
The chart indicates that it would take a phenominal amount of cigarettes in a confined area to reach toxicity. Exposure to toxins is not a good thing, so how bad is too bad? What you and I can tolerate without discomfort, may hurt another person.