They were there long before us, and they'll be there long after we're dust. Why worry?
Interestingly, the chance of the average American dying in in a meteor or asteroid impact is about the same as the chance of an average American dying in an airliner crash. (Sounds ridiculous, but think about it carefully.)
The problem with "us" is that "us" hasn't been around for very long. And a lot of life forms before "us" weren't around after large impacts.