Even programming in construction detours would be a challenge.
While a lot of drivers out there are pretty scary behind the wheel, they retain the potential to react appropriately in sudden situations despite ordinary inattentiveness or poor skills.
A computer might not be able to do so, especially if it is installing one of bazillions of updates and security patches, etc.
Of course, compromise such a system with evil intent, and the potential for carnage is phenomenal, not to mention the potential for the 'removal' of individuals, while blaming a 'computer glitch'.
It'd make a great sci-fi plot, but I can't honestly say I see where it would be an improvement.
You are right. There are a myriad of unique situations which a human can [usually] handle, but a computer will not be able to. Eventually, they might get the software right for 99.9% of those, but that pesky 0.1% will be enough to generate tons of horror stories. That and the inevitable lawsuits will ruin a lot of these companies early on. Software products typically have NO warranty. They will not be able to get away with that for software that controls driverless cars.