Have you considered that they walked away, because they are the rightful owners and do not want to see it destroyed?
That would be Biblical, would it not?
I think it was more like the misguided belief that if you are nice to evil people, they will be nice to you.
I’ve been there. There is nothing to destroy except the Al Aqsa and the Dome of the Rock. I have no objections to their destruction.