True.
However, attacking a person under such circumstances is an appropriate technique, although I guess as you say it’s not really a fallacy in such a case.
What I am suggesting is not just pointing out that they are using a fallacy, it’s destroying the validity of their authority, when that is appropriate.
validity implies a conclusion that necessarily follows that can not be false if the premise is true.
Authority is an immaterial issue. Notice I didn't say irrelevent, because it may be relevent, its not germane to refuting an argument unless the argument is dependent upon an appeal to authority, and the person making the argument is claiming to be an authority.
Its sufficient to refute the conclusion by pointing out flawed reasoning, which essentially demolishes the argument. It is incumbent upon the asserter to come up with a new syllogism that is not fallacious.
Far better to acknowlege a valid conclusion and then attack it as being unsound, i.e., either of the premises are actually untrue. An attack on soundness essentially boils down to acceptance of facts.
People who have no discipline in logic usually resort to ad hominem when confronted with touche' on their argument. The first person resorting to ad hominem in a debate loses, or at the very least is evidentiary of their tenuous position. When pressed in such position, shallow thinking people usually implode and resort to vulgarity.
That's a sure sign of a small vocabulary and weak mindedness.
We may just have a differnce of semantics. I wouldn't characterize it as destroying the validity of their authority as much as destroying the idea that their authority validates the argument.