You should look into getting a family card that has both of your names on it. You surely wouldn't expect your husband to withdraw money from a bank account that was set up only in your name. If you want dual privileges it must be set up that way.
We're talking about library books here, not a secret Swiss bank account. Geez. He said, "as long as you're going to the library, would you mind picking up my books at the hold desk." When I got home he said, "where are my books?" I said, "the library won't give them to me because it's an invasion of your privacy." He said, "Son of a bitch! What kind of anal control freaks are they over there?"
BTW, since then we have filled out the paperwork and signed the release forms that "allows" us to pick up each other's books so the librarians don't have to worry about "invasion of privacy" when checking out the latest Vince Flynn thriller.