You are right. I should have just walked off and said "see ya later." Such stupidity isn't worth the trouble. Fortuantely, I rarely ever see this guy, and now that I know where he works I can avoid him.
The best way to handle such moonbats is look at him with a great expression of concern and say " you aren't suppose to now that" Turn your back and punch some numbers into your cell phone. Act like you are being told how to handle the situation. When you get off the phone turn to your friend and say " That was close, I gotta be honest maybe you should stay in your house for awhile". Then leave.