Document every single interaction with him. Time, place, conversation exchange, etc. If at all possible, always have another person present...even in any lunch room, break room.
What I’m reading is that you quite literally feel “unsafe.” Perhaps you should report this to HR. Obviously they care about such things.