Many accidents are caused by negligence. There are millions of accident scenarios and it is impossible to think of all of them.
No law or society can prevent accidents. To think so is negligence in itself.
Yes, rarely there is a true accident that would warrant no punishment. However, in the scheme of things these are a very small portion of car crashes causing damage or death. Take a look at your news and watch how many crazy "accidents" go unprosecuted and you'll get a feel for where I'm coming from.
A couple of times a week in my area we get crazy stories like a lady driving on the sidewalk because she was having a bad day, a driver falling asleep and slamming into a police car, etc. These accidents go unprosecuted. Why? Are they not caused by negligence? Well, yes they are, but they don't involve alcohol so they are just normal "accidents."
Hitting a deer would be one of the rare accidents. Some weather conditions would also warrant being classified as accidents.
If prosecution of negligence was more aggressive, my bet is that insurance rates would go down.