I think if the X-15 had been carried to its logical conclusion, we would have SSTO today. Water under the bridge now. Also NERVA should never have been canceled.
I don't know enough about it to have a serious opinion, but I've been told the same thing by some very, very well informed Air Force types. They blamed Kennedy for misdirecting the effort to the moon (for narrowly political reasons, of course) and setting the conquest of near space back for what is now two generations and counting.