Yes this is true we didn't have a large enough rocket back in 1962 to launch to the moon but we did at the end of the decade.
Saturn V was an American human-rated super heavy-lift launch vehicle used by NASA between 1967 and 1973. A three-stage, liquid-fueled rocket, the Saturn V was developed under the Apollo program for human exploration of the Moon and was later used to launch Skylab, the first American space station.
How easy would it have been for von Braun to say, "When Saturn V comes fully on line, we'll be able to go to the moon!" When he talked about Saturn V he was not keeping it a secret. His job as architect was already essentially a completely done deal.
Saturn V was only used for lifting into Low Earth Orbit, as Apollo was and as Skylab was also.
Is your answer of what Armstrong was asked, "Why haven't we gone back to the moon in over 50 years?" anything like Neil's first-hand-knowledge answer?