In a way I see your point - but those places (government offices) are paid for by everyone and at some point in time must be visited by everyone.
The problem is that the anti-smokers feel it is within their right to not be inconvenienced by smoke on the off chance they may some day possibly decide to step foot in someplace. They therefore ignore the fact that the only one paying for that place is the person that owns it.
I remember when the management company that owns several of the malls in the state decided to make them non-smoking, with the exception of restaurants that had smoking sections. This was no big deal to me as I'm not a big mall shopper and there were still other malls. This was before statewide restrictions.
When statewide smoking restrictions were being proposed a couple years later, that company was at the forefront of including all malls. Apparently their policy had been hurting business and they wanted to "level the playing field."