Along those lines, does anyone know of a program that will crawl (scrape) a website of all documents? Automatically? With storage space so cheap, I’d do that to so many websites.
Sorry, unable to help you here. It is Web Archive and the Wayback Machine when a long ago site wishes to be seen. There are times when it cannot even be found there, depending on certain conditions.
Who knows, 10 years from now under UN supervision, our internet will exist of no more than a tin can and string held together by spider webs and NSA doing their part by producing more corn to can in order there be enough tin cans for everyone.