Free Republic
Browse · Search
News/Activism
Topics · Post Article

To: RightField
robot.txt is a file at your website that tells automated web browsing systems called robots (like the ones at Wayback) whether they may or may not access the site.

Facebook seems to generally exclude robots from accessing their site, preventing the wayback machine from archiving facebook pages. Your own facebook profile would similarly not be available on the Wayback machine.
97 posted on 06/14/2017 3:35:18 PM PDT by Jagermonster (TANSTAAFL)
[ Post Reply | Private Reply | To 86 | View Replies ]


To: Jagermonster

Interesting. Thanks for that information.


100 posted on 06/16/2017 8:30:21 AM PDT by RightField
[ Post Reply | Private Reply | To 97 | View Replies ]

Free Republic
Browse · Search
News/Activism
Topics · Post Article


FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson