It is part of the Roman Wall. Jesus said there would be nothing left of the temple.
So is the retaining wall of the platform the temple sat on "the temple"?
The reason people dismantled the temple ruins after the Romans torched it was to recover all of the gold that had melted and flowed down between the stones. The retaining wall would not have had melted gold between the stones.