Don't use a 'bot' program to filter through the content on the website. Okay?
Posted on 11/02/2004 11:52:22 AM PST by Momaw Nadon
What is crawling/spidering?
Just wondering the same thing....what's the Geek Gibberish?
I was wondering that myself...
It's a WWW bot thing. Has nothing to do with you, just disregard.
Yeah... if you all are gonna tell us not to do something... can you at least explain what it is you dont want us to do? Crawling/spidering?? what kind of jargon is that?
I think that means trolling!
This has made me paranoid. I thought it might just be me. Like maybe I hit reload to many times. Creepy. I sent a private request for info on it.
Don't use a 'bot' program to filter through the content on the website. Okay?
bump for reading later
no, it doesn't.
robots.txt puts limits on what pages the web search engine 'spiders' can read and catalog. No robots.txt or spiders ignoring it = all pages spidered, big use of resources.
Thank you for asking. I've no clue what it means. Sounds creepy though, doesn't it? Spidering...icky.
Methinks FR is sending a message to some pernicious hackers...
ENOUGH WITH THE VANITIES!!!
I guess I don't have to worry, since I have NO CLUE what a "bot" program is.
Folks like google, using computer programs to read and index web pages.
Every major site has a robots.txt page that tells spyders which pages to index and which ones to skip.
Robots can also put a drag on a site by requesting lots of pages. It slows the site down for everyone. A malicious robot can deliberately attempt to slow a site down. Thi is called a Denial of Service (DOS) attack.
It's a Peter Parker thing......
It's an automatic program (called a robot) that follows all the links and records all the content it comes across (a process called indexing). It's how search engines like google finds the content that you search. The Robots.txt file tells robots what's permissible to access. All the indexing takes bandwidth from the site, so some sites are very restrictive on how much a robot is allowed to download.
Search engines "crawl" or "spider" the internet looking for sites and pages to add to their database.
Actually, I think it's just some remote program that is trying to access FR and the mods are telling whoever is doing it to knock it off.
It means having a program walk through your web site.
Either they are indexing (like google) or data mining for info.
Robots.txt is a file made available for such programs which is supposed to tell these programs what the rules are.
Being firewalled means configuring the hardware so that the machines doing this can't get access to Jim's servers.
"ENOUGH WITH THE VANITIES!!!"
I would sware that is all some people can say.
Another reason to restrict robots here is that the Dems apparently are using FR content in order to find attack material on conservatives. We had someone here who had some of his postings thrown back in his face in a political race (I think). I forget the details, but others can probably supply them.
Dear god.
its an automated search program that tracks what is on a website
Web crawlers are used by search sites like Google.
FReepers rule. I have no idea what the heck everyone is talking about. My knowledge of computers consists of barely recognizing the letters and commands on my keyboard.
The rest basically means that if someone is trying to do something like cyber attacks, and going everywhere on FreeRepublic through automated entrance to grab info, they will be blocked out (firewalle).
This is not a warning, but just stating what happens if someone decides they are going to attack this website.
Posters do not have to worry. Folks who can throw multiple DS3's at Freerepublic have to worry that they may be guilty of abuse.
On election day, the Democrats may try to trash the site.
Also, news agencies, because of their inability to find genuine info, may be trying to rely on Freerepublic as a good source of news (lame stream media).
It was that book along with "Stolen Valor" and "Fahrenhype 9/11" that swung my vote back to Bush from Badnarik. I could not live with myself if I contributed to a Kerry victory. I cannot trust the character of the man...though I think W. has a good character and is a far better man than Kerry could ever pretend to be.
Don't get me wrong, I'm pretty disgusted with Bush on his domestic agenda, but Kerry running foreign policy would be a nightmare exceeding even Clinton. And foreign policy is what is most important right now. "Global Test" my ass!
bots/spiders are how search engines get their data, they put a robots.txt in their files to keep the bots/spiders away today (to keep the site as fast as possible)
Ooops, last post was meant for you...linking in reply is messed up...
For an example, see here. It's the Teleport Pro site.
I've been using Teleport on and off for years. But never here:)
These things can be real bandwith suckers since they can spawn multiple processes that suck as much as they can from a web site. They usually also allow you to "overlook" the robots.txt file.
Does that mean something like Google can't or isn't suppose to index the site?
robots.txt just gives the guidelines - whether the spiders obey it is up the the people who run the spiders.
I'm not sure what FR has for settings, but google does index free republic.
"use a 'bot' program" ???
Heck, I'm still trying to figure out how to post a great Zot picture I have. HTML and me do not agree.
I'm using Wanadoo in Spain and when I first accessed Free Republic this morning, I got hit with 6 viruses (it's a good thing I have a firewall). I don't know if it had anything to do with what was posted.
Dan
Biblical Christianity web site
Biblical Christianity message board
oh, i don't know. this being election day, i am a bit frazzled and when i get frazzled, i usually take it out on the dog. my dog is greatful for all the vanities that have popped out of the blue today.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.