You are correct that one option would be to depart, if they would let you do so.
Another option would be to stand your ground, in defense of yourself, family, or other present innocents.
It is taught in CCW classes that sometimes all it takes to deter a threat or attack is presenting the gun. So that is also an option, and if one felt fear of bodily harm would be warranted, but it also has risks all its own.
I would be most concerned about where the muzzle was pointed while he was 'assessing' the changing circumstances in front of him. If he felt the threat was enough to present the gun but not enough to justify shooting, then a 'sul' ready position might have been smarter, keeping the gun pointed down while very ready to point in if necessary.
Other than that, I think the old adage applies...
