That sort of thing can happen both ways. At my local P.O. it seems they are intentionally putting the least English-competent people possible at the service counter. When my elderly Mom asks some of them to repeat what they've just said (her hearing is perfect....it's the lousy diction, heavy accents and brusque attitudes of the clerks that's the problem) they smirk and merely speak more loudly, as if to say "there's nothing wrong with me, it's you that's the problem". They're arrogant because they know nobody will dare suggest that they improve their English skills for fear of being called "racist" and their Union will protect them from any customer complaints.
And you're right -- I don't think anyone dares to complain about someone working at the Post Office. Because, you know that precious little package you just handed over to them? Well, it would be a real shame if something were to happen to that package after you walked out the door ...
They are untouchable. They know it. If you cross them, they will get you back, and they know that they will win that little game.