Depends on the pool, doesn't it? The right to use a public pool might be extended by a city government. The regulations defining the manner in which such rights may be enjoyed are the subject of federal, state and local law and regulation.
Where would you contend that the right to use a public pool comes from, if not society?
Where do rights come from and how are they defined?