I would have waited for him at the place you ate....and if not possible sat in the car.
It was after we left restaurant. He would never have assaulted me. He knew it meant my early meeting next morning meant I had no interest. Had contact with him 15 years later he bought the company I worked for. Glad we parted on good terms.