My library now has a link on their webpage where you can download audio books for free to your MP3 player. :) So I don’t think what you’re doing is wrong because the libraries provide the service.
My library doesn't, yet, but most are moving in that direction. At the very least, libraries should let folks download any work that's out of copyright -- many of them are already online, but librarians could help folks find them. It's kinda what they do.