To the extent such decisions are made matters of legal regulation, the determinations are made by the legislative process subject to judicial review.
Now, where do you contend that the right to swim in a public pool comes from?
That isn't the measure. That's just the process which takes place after the determination is made. But, you knew that. I asked by what measure does one go about determining which activities should be prohibited?
...where do you contend that the right to swim in a public pool comes from?
I contend that no such right exists.