I’m betting 99% of the reason was to keep the research out of Russian hands. The Arrow was designed to intercept Russian bombers coming in over the remote regions of the arctic.
At that very moment, we thought -we- might need to send bombers into Russia over some very remote regions of the arctic. We didn’t want our B-52s and B-58s to run into a Russian Avro Arrow clone. Remember the Soviets copied everything they could back then, so that was a very real possibility. And the Arrow was almost -purpose built- to defend the USSR against SAC bombers.
It would have been like a very high quality MiG-25.
Politics. The Pentagon will cancel an aircraft program and ‘order the tooling & jigs’ destroyed so that the program cannot be resurrected by a future Congress. This is how the senior generals prevent the canceled program from competing with a newer, more favored program for funding. Case in point: The F-22 vs. the F-35.
In a lot of ways the F-22 is the superior air frame. If Lockheed Martin were to graft some of the more advanced features of the F-35 onto an F-22B, you’d have the ultimate pure fighter. Lockheed Martin won’t suggest it because they control both programs. But if the F-22 was built by a competitor? Bet your bottom dollar that the tooling would have been destroyed by now.
Diefenbaker wanted to destroy the Arrow and, in case it was the wrong decision, he wanted no memorials to exist.
There is a small bright cloud here. Many of the suddenly out of work aeronautical engineers drifted to the US and signed on with a new government agency NASA. It could be said that if it wasn't for the immigrant engineers from Germany and Canada the US would have had a much tougher time in the space race.
Seems sensible. If the value of interceptors for you has dropped and the enemy still has need of them, then dunking the program will slow their progress.