I don't worry so much about a dirty bomb as I do about nuclear littering. A bomb advertises itself. But what if they just dropped the stuff in a high-traffic area?
It's just poison. Some will get the needed treatment/decontamination in time, some won't, and the rest of the nation won't even realize that much of anything has happened.
No, the big danger from such things is that the news media blows it out of porportion and causes some big scare...and that the resulting scare does real damage.