Yup. And there's really nothing we can do about it. So, again, why worry?
Eh, there are possible ways we could do something about it. It's a bit more complex than simply nuking something incoming, but it may not be an insurmountable problem.
I'd hate to have civilization end and not have explored every possible way to avoid it.
If we get enough fo a lead time, we CAN do something about it.
The NEAR-Shoemaker probe landed on Eros and transmitted scientific data for 16 days after landing. If an unmanned spacecraft carrying scientific insturments can land on an asteroid - an unmanned spacecraft with a "physics package" can land on an asteroid.
Get enough lead time, land the craft on the asteroid, and then activating the "physics package" should make the problem asteroid go away.