First of all, one has to be careful with the 'Ready to pass away' thing - According to Yeshua, it cannot pass away until every jot and tittle of Torah AND the Prophets is fulfilled. That time, according to Yeshua in His Revelation is unarguably at the moment of the last trumpet:
Rev 10:7 But in the days of the voice of the seventh angel, when he shall begin to sound, the mystery of God should be finished, as he hath declared to his servants the prophets.
And it may take much longer than that - As Torah and the prophets go well beyond the Kingdom... But the mystery given to His prophets is finished at that time, so one would at least have to get to there.
As to the Temple, the problem with your position is that it ignores Ezekiel's Temple, described in great detail, with sacrifices and processes that have never been performed before... That temple, where the whole world comes to (on the three Holy Days that require it)... That Temple, where the Prince who will come will take his breakfast, on the porch of the Eastern Gate... The Eastern Gate, which has remained shut for the exclusive use and purpose of that Prince. That same gate where He will arrive in power and glory after touching down on the Mount of Olives...
I know that it is a tradition among Christians that the 'triumphal entry' fulfilled that last bit, but such a claim is made in ignorance - The triumphal entry had to have happened at the northern gate - the way to Bethlehem.
No, that Eastern Gate must be there for His arrival - And if it is there, so then too must the Temple be there. Until every jot and tittle is done, it will certainly come to pass.
Good mention of Revelation 10. Thanks.