Only the kool-aide fans will stick around if he gives away one of the key pillars of his primary victory.He pummeled his primary opponents and Hilly using this promise, differentiating himself from the others to be trustworthy on this promise.
He made it worse with his responses immediately after the meetings, when he said that the wall will be built later, and that they are re-furbishing the existing fence.
Ivanka and John Kelly probably told him that the voters have forgotten and moved on. He does not have any conservative advisors anymore like during the campaign.
It is Trump who wants the Wall!