Posted on 11/02/2004 11:55:11 AM PST by sonsofliberty2000
Internet search engine syper and crawl the net indexing text and pics into thier search database. They appear to the FR server like a normal user but hit every link in the site and create quite a load on the server. The robot.txt file contains instructions for the spyder/crawler on which areas of the site are OK to index. I'll bet today that the robot.txt tells search engines not to index the site today. Now search engine are suppost to obey instruction in the robot.txt. But not all do, FR is letting them know that FR will block thier IPAddress from indexing FR if thier ignore the robot.txt request not to index today.
I got 15 phone #s for FL. I'm in Iowa. I have received plenty of pre-recorded calls but no live person calls. The list is coming from BC04 and they have check boxes next to the names so they don't get called multiple times (I hope).
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.