Don't use a 'bot' program to filter through the content on the website. Okay?
Posted on 11/02/2004 11:52:22 AM PST by Momaw Nadon
What is crawling/spidering?
Just wondering the same thing....what's the Geek Gibberish?
I was wondering that myself...
It's a WWW bot thing. Has nothing to do with you, just disregard.
Yeah... if you all are gonna tell us not to do something... can you at least explain what it is you dont want us to do? Crawling/spidering?? what kind of jargon is that?
I think that means trolling!
This has made me paranoid. I thought it might just be me. Like maybe I hit reload to many times. Creepy. I sent a private request for info on it.
Don't use a 'bot' program to filter through the content on the website. Okay?
bump for reading later
no, it doesn't.
robots.txt puts limits on what pages the web search engine 'spiders' can read and catalog. No robots.txt or spiders ignoring it = all pages spidered, big use of resources.
Thank you for asking. I've no clue what it means. Sounds creepy though, doesn't it? Spidering...icky.
Methinks FR is sending a message to some pernicious hackers...
ENOUGH WITH THE VANITIES!!!
I guess I don't have to worry, since I have NO CLUE what a "bot" program is.
Folks like google, using computer programs to read and index web pages.
Every major site has a robots.txt page that tells spyders which pages to index and which ones to skip.
Robots can also put a drag on a site by requesting lots of pages. It slows the site down for everyone. A malicious robot can deliberately attempt to slow a site down. Thi is called a Denial of Service (DOS) attack.
It's a Peter Parker thing......
It's an automatic program (called a robot) that follows all the links and records all the content it comes across (a process called indexing). It's how search engines like google finds the content that you search. The Robots.txt file tells robots what's permissible to access. All the indexing takes bandwidth from the site, so some sites are very restrictive on how much a robot is allowed to download.
Search engines "crawl" or "spider" the internet looking for sites and pages to add to their database.
Actually, I think it's just some remote program that is trying to access FR and the mods are telling whoever is doing it to knock it off.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.