When my boss early in my bus career became CEO of a fortune 500 company, I had dinner with him while in a business trip to CA. Then he wanted to go back to his apt to change his shirt. When he took off his shirt in front if me in the living room, I was grossed out and I ended the evening.
I would have waited for him at the place you ate....and if not possible sat in the car.