The HOA does not own any of the property, and hardly ever holds a meeting. The trespassers get up in the faces of the people on whose grass they are treading, bringing weeds out of the woods onto the lawns.
One of these days there will be an incident. Then the courts will rule in favor of the trespassers, and the insurers will punish the homeowners who were trespassed upon.
The "freedom to roam" is a Nordic thing, even codified in law, but not recognized in the United States.
We didn't have that in Ireland, son.
The second part of your post is what matters.