That is exactly what a weirdo would like you to do.
Be aware, be alert, be cautious, but NEVER let some crazy creep drive you away, it is exactly what IT wants.
You are right, of course, “never let the bastards win” is a proper course to set.
However, if my child was in the equation then her safety would be more important than anything else to me.