Sunday Afternoon Computer Question: How to Archive a Website?

Me

[Paul R.]

A relatively small web forum I am a member of is shutting down soon. There are a lot of posts on it I and other members would like to archive to HD for reference. Saving individual pages is VERY time consuming, and we only have until the end of the month. Do any of our FReeper computer guru's have any experience with this?

[Paul R.]

I attempted to save the site with a program ostensibly made for that purpose, HTTrack, but get a mirror error. So far I'm unable to find a cause for the error - not that I know much about such errors. We only have until the end of June to find a way to archive the site. There has been some discussion of doing just that on a thread on the site, but none of us have any experience with this.

So far, at least, the forum "master" has not raised any objection to saving of the material for personal use, but is also not assisting and may not have any time or ideas anyway.

Do any FReepers have any experience with something like this? Maybe try some other archival program?

Thanks in advance!

[Jeff Chandler]

Is this a database driven website?

We need more info.

[E. Pluribus Unum]

wget might work.

https://eternallybored.org/misc/wget/

[Jeff Chandler]

https://www.google.com/search?q=httrack+mirror+error&sca_esv=83e1b8e3588a8f5f&rlz=1C1AVFC_enUS779US780&ei=SW9vZvSrLp3GkPIPu_aroAE&ved=0ahUKEwi09PzeneGGAxUdI0QIHTv7ChQQ4dUDCBE&uact=5&oq=httrack+mirror+error&gs_lp=Egxnd3Mtd2l6LXNlcnAiFGh0dHJhY2sgbWlycm9yIGVycm9yMgsQABiABBiRAhiKBTIGEAAYFhgeMgsQABiABBiGAxiKBTILEAAYgAQYhgMYigUyCxAAGIAEGIYDGIoFMggQABiABBiiBDIIEAAYgAQYogQyCBAAGIAEGKIESIEeUPUHWLkccAF4AZABAJgBZaABxAiqAQQxMi4xuAEDyAEA-AEBmAIOoALdCcICChAAGLADGNYEGEfCAg0QABiABBiwAxhDGIoFwgIFEAAYgATCAgoQABiABBhDGIoFwgIIEAAYFhgeGA-YAwCIBgGQBgqSBwQxMS4zoAeNRg&sclient=gws-wiz-serp

[E. Pluribus Unum]

https://learn.microsoft.com/en-us/answers/questions/1167673/how-to-use-wget-command-on-windows-(for-recursive

[so_real]

Archiving Websites with Wget

A second vote for wget. Good luck!

[Theo]

Paul,

I’d be happy to help. I’ve been doing website stuff for over 30 years.

The simplest solution is to just use SiteSucker. Easy to use, but make sure it’s configured right or it could end up saving webpages from other sites connected to your site.

Is it a WordPress based site, or some other kind of forum that consists of files and a database?

Let me know how I can help.

[zeebee]

+1 for Sitesucker

[Paul R.]

I don’t think so. It’s just posts about fishing, fishing gear, etc. Plenty of pics.

It’s my go to source for info. and advice regarding non-Chinese gear, which I avoid when possible, any more.

[Paul R.]

This is not my site. But, again, the webmaster has not expressed any objection to saving of the pages of the site for personal use, and he HAS been involved in the thread about the shutdown. Multiple members have been saving individual pages with copy/paste, etc. But even for a small forum that’s not practical to save the whole thing or maintain any structure.

It’s ALMOST as if I heard FR was shutting down and I wanted to archive it, except that FR is MUCH, MUCH bigger. (Probably 10,000x if I had to guess.)

SFAIK it’s not a WordPress type site, but I am not sure. It IS a forum.

[Pollard]

If it’s a forum, 99.9% chance is has a database for storing posts. Most any website scraper/down-loader will just grab the html that a browser gets from that and save it as html pages, plus images.

I’m on Linux/Ubuntu and I’ve used httrack for that but like someone mentioned about a different tool. You have to be careful what depth of links you grab. Might need 2-3 depending on how the site works. If the forum allows embedded youtube videos it could get big unless you figure out how to filter that out.

Like everything tech the answer is, it depends on some pesky thing like variables.

Plug the url into builtwith.com and you might get an idea what it’s ‘built with’. At any rate, you’re only going to be able to scrape the html pages that get rendered for the browser.

[Paul R.]

I guess I’d better add that I’m using Windows 10 or 11...

[bankwalker]

Go to the URL of your site and add “/wp-admin” to the end. For example, if your site is www.example.com, you would go to www.example.com/wp-admin. If you get a login page it is a wordpress site. If it is then there are wordpress plugins that do full backups.

[freeandfreezing]

One quick thing to check is how much if any of the site has already been archived by archive.org’s wayback machine.

[Paul R.]

Ok, I tried that (adding the “/wp-admin”) and got a “forbidden access” error message.

So I guess that confirms it’s not a wordpress site.

[Paul R.]

I found it, but it seems to be bits and pieces. I am NOT sure I’m using the archive.org site correctly. It seems almost like another planet.

[PAR35]

Anyone considered offering to buy the site and content from him?

[ptsal]

Bkmk

[Jeff Chandler]

What is the URL?

[Paul R.]

A few of the members have been talking about that possibility, but, apparently it isn’t going anywhere. The site is one of ~60 forums owned by the same company and apparently the whole thing is going down shortly.