Many of the casing stones fell off due to a great quake in 1303. The ruling sultan of the time hauled the stones off to rebuilt parts of Cairo.
A lot more were removed by Muhammad Ali Pasha during the early 19th century to build a mosque in Cairo.
He, BTW, was a secularizer, by the standards of his time and place, not an Islamist fanatic.
I stand corrected, thanks!