Who's "we"? You and I might agree but there's a whole bunch of other people who need to agree too. By "touching" I assume you mean private parts, but I've been unexpectedly touched elsewhere, sometimes hugged, more rarely kissed. I didn't take offense but that doesn't mean others wouldn't.
As to "exposing ones self", you only mention men. What about the woman in a low cut top who bends over and provides a clear view of a well formed breast? Or the woman in a short skirt who allows a clear view. You might not object to that. I might not object to that. Can we get everybody to agree with us?
I really am not concerned about wolf whistles or comments, and I don't believe the law is either...
Some people are concerned about wolf whistles or comments. The concern or lack of concern of you and me is not the only consideration.
As to the law, from the US Code (emphasis added):
§ 1604.11 Sexual harassment.(a) Harassment on the basis of sex is a violation of section 703 of title VII. 1 Unwelcome sexual advances, requests for sexual favors, and other verbal or physical conduct of a sexual nature constitute sexual harassment when (1) submission to such conduct is made either explicitly or implicitly a term or condition of an individual's employment, (2) submission to or rejection of such conduct by an individual is used as the basis for employment decisions affecting such individual, or (3) such conduct has the purpose or effect of unreasonably interfering with an individual's work performance or creating an intimidating, hostile, or offensive working environment.
...not in eyes of the law.
Yet.