[John Robinson]

Well, I had more in mind some type of program that would automatically document changes to an article's words by comparing a saved copy with a freshly downloaded copy. Any changes to the document would be made public. And some way to find the URL of articles if they've moved. More than likely, these spiders would have to be custom built for each site and each format change of those sites. A non-trivial task which would end up a community programming project. Spiders aren't very difficult to write, but hundreds of spider programs for sites with varying structure could consume thousands of hours.

[mrsmith]

"hundreds of spider programs for sites with varying structure could consume thousands of hours. "

Have you considered a distributed program like that "seti search" one? One that freepers could run on our computers?

[GovernmentShrinker]

I'm a complete non-techie, but I've wondered if there's a way to have people post an article with a link, and have only the link show for as long as it's working; then the first time the link retrieves some sort of error code, have the full text post kick in permanently. I don't know if it's feasible, since different sites produce different types of responses to expired links, making the programming for a recognition system difficult. Alternatively, the system could rely on freepers, with a standing instruction to click the "insert full text" button if and only if the link doesn't work. If something like this could be set up, I think few sites would have any objection to the permanent archiving -- most just want the hits for the few days the articles are accessible on their sites, to show to their advertisers. Few of the news sites are deriving any significant income from paid archive retrievals by individuals looking for a specific article. Just a thought to keep in mind if problems of this sort crop up again in the future.

[RedWing9]

More than likely, these spiders would have to be custom built for each site and each format change of those sites.

Spider's give me the willies :o)