You say "A dirty bomb would have to be very well put together to be effective."
Here is one article that points out that a dirty bomb attack could (although probably not) leave the ground zero unsafe for decades:
'Dirty bomb' threatens U.S. with near terror attack
By Bill Nichols, Mimi Hall and Peter Eisler, USA TODAY
"In terms of how large an area would be contaminated by radiation, the damage would depend on the type and quantity of radioactive material. The contamination could dissipate in weeks, or it could render an area uninhabitable for decades. Abnormal levels of radiation would be spewed tens of miles in a worst-case scenario. More likely, experts say, the evacuation area would be a few dozen city blocks...
In one scenario studied by the federation, the detonation in Manhattan of a dirty bomb, using a piece of cobalt stolen from a food-irradiation plant, would spread contamination over an area of about 300 city blocks and render most of the city uninhabitable for decades. However, federation scientists note that obtaining such a potent radiological source and using it effectively would be very difficult.
http://www.usatoday.com/news/nation/2002/06/11/bomb-usat.htm