You are right. If the building had been properly constructed, that ANFO blast shouldn't have done the damage it did. However, a shoddily built Federal Building, as this one appears to have been, could have been taken down by a lesser explosive charge than one would expect to use on a properly constructed building.
The haste in tearing down the remnants of this Federal Building could have been to hide the fact the tax payers had been ripped off and someone's "brother-in-law" made a profit from shabbily constructing the building in the first place.
Conspiracy theories sometimes make more sense if you look for a "greed" motive rather than some other nefarious plot.
Very good point.