He may not have even realized that people left. The complaints were made to the production manager, not him. I’ve been on a couple of movie sets during filming and there’s a lot of people around. The stars don’t mingle with people they’re not directly involved with. So why would he notice that 3 or 4 people left? And if he did, maybe he wasn’t told the real reason. Don’t want to upset the ‘Star’, we’ll tell him later.
I’ve been on movie sets too, with the producer there. And especially with this sort of small-budget films, you don’t have an on-site producer and actor unaware of accidental discharges or a staff walkout with replacements hired.