I'm not an early adopter either, preferring to let others with more time on their hands do the trail blazing, however when the dust settles years from now the cost for drivers insurance will be about triple if you want to continue driving yourself.
Put another way, car insurance for driverless cars might be really cheap because they don’t get into accidents.
Its even possible that manufacturers will include the insurance as a added feature. Think drive-train warranty.
I think those cars will get a lot of people killed if there is no manual override. But if people become dependent on the cars, they won't develop the skills they need anyway.
Consider, those of us who grew up with no power steering, manual transmissions, no power brakes, no airbags, steel dashboards, no seat belts, no automatic braking, no rearview camera, no lane sensor, etc. learned to do all that on our own as a part of driving, in fair weather and foul, and learned to drive defensively, to avoid problems, and always (when possible) leave ourselves an 'out' if drivers in front of us had a problem.
Those skills (braking on ice, for instance) have been by and large, relegated to computers, and the results aren't exactly stunning.
Even with all that supposedly lifesaving stuff, there is still a slaughter on the highways.
One more gizmo isn't going to stop that, and for a time, in some instances, may well make it worse.
I'll pay the extra if I must, but I'll drive.