I have no doubt driverless cars can work. But people are viewing them as simply technological constructs, completely unconnected to the society in which they operate.
For example, in my city, if driverless cars passed through certain areas, they would most definitely be objects of graffiti “tagging” at best - at worst they would be constantly vandalized, seemingly for fun, or broken into. So then you need to upgrade their protection or laws regarding tampering, etc... Overall, beyond the simple question of “will they work” - these devices then enter the social, economic, racial, environmental etc.. spheres, where technology will take a back-seat.
Driverless cars will NEVER work for the very reason you're alluding to in your post: They can NEVER account for all the variables in human behavior, nor can the A.I. used by them adapt like the human brain to be able to react like a human being to avoid an accident.
Were that true, we would see ZERO accidents involving driverless vehicles either now, or in the very near future. The problem is driverless cars and A.I. are still created by human beings and we're flawed. We do not have perfect logic, nor do we always react in logical ways.
The best A.I. will never be able to account for all the different combinations of reactions that we human beings are capable of producing and reacting to.
Never.