Retailers used to accept cards because it encouraged business and more potential spending. Restaurants are a bit different I suppose, you know you are going there in most cases and if you know the policy you bring cash or pay the fee. Personally it does bother me and I reduce my tip from a approx 22% ish down to 18% shifting the cost to the server which is unfair, but so is charging the extra 3-4%.
VISA and Mastercard used to ban these extra charges. But that must have changed as I have seen it in several small restaurants recently.
Its kind of weird in a way. I am also seeing more and more businesses NOT take cash and only take cards, so it is hard to know what to make of the current business models.
I used to process cards for a business which did not add the transaction fee to customers who paid cash. Customers who paid with a card paid for the card fees. I like businesses which do that.