A taxpayer-supported gym isn't a good analogy, as even poor people can get their physical fitness for free (push-ups, sit-ups, jumping jacks). Books are not free.
That said, taxpayer funding for libraries only comes in the form of matching funds -- the government will not pay for libraries in communities that refuse to contribute private support.
Gym equipment helps you to work out your body as books help you to work out your mind.
A lack of public libraries would not stifle your own freedom of speech. You would still be at liberty to say what you want.
You could even go to a cafe or bar or the park (or an online forum like FR) and exercise your mind talking with other people.