I would have let him see that I had a gun and you might have been saved him following you later. Usually just the sight of the gun will cause the bad guys to go away.
As long as you don't point it at them it's not assault, at least in my state.