One of the things agreed to was that he would not only halt production of WMDs, but destroy any that he had. An inspection regime was put in place to verify that he was doing what he said he was doing.
As example, he and his government admitted to having certain quantities of Anthrax. His responsibility for that is to allow the inspectors access so that they can observe and certify that X amount of Y material was destroyed. Instead, these materials have disappeared with no explanation. It's up to Hussein to explain what happened to the materials, and cooperate so that the inspectors can certify - else there's no real justification, other than laziness on our and the U.N.'s part, for NOT invading and removing his government from power.