A dirty bomb would have to be very well put together to be effective. Some radiological material simply isn't radioactive enough to be a threat - and for ones that are, they still need a good deal of it, which makes creating, moving, and setting it off a complex operation.
Hiroshima didn't depopulate even after we gave it a lot bigger dose of radiation than a dirty bomb can produce.
You say "A dirty bomb would have to be very well put together to be effective."
Here is one article that points out that a dirty bomb attack could (although probably not) leave the ground zero unsafe for decades:
'Dirty bomb' threatens U.S. with near terror attack
By Bill Nichols, Mimi Hall and Peter Eisler, USA TODAY
"In terms of how large an area would be contaminated by radiation, the damage would depend on the type and quantity of radioactive material. The contamination could dissipate in weeks, or it could render an area uninhabitable for decades. Abnormal levels of radiation would be spewed tens of miles in a worst-case scenario. More likely, experts say, the evacuation area would be a few dozen city blocks...
In one scenario studied by the federation, the detonation in Manhattan of a dirty bomb, using a piece of cobalt stolen from a food-irradiation plant, would spread contamination over an area of about 300 city blocks and render most of the city uninhabitable for decades. However, federation scientists note that obtaining such a potent radiological source and using it effectively would be very difficult.
http://www.usatoday.com/news/nation/2002/06/11/bomb-usat.htm