Free Republic
Browse · Search
News/Activism
Topics · Post Article

To: jurroppi1

I did just check http://obamaphone.net/robots.txt and there is a robots.txt file there. It is minimalistic and uses some tags that are not contained under the original development of the standard. They are as follows:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

Much more info is available at: http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html

Doubtless there are several other tags and methods to deal with bots of this nature and to impress control over how they index a given site.

follow the links and read (there are x-robots tags as well as other meta tags that can be used to force a bot’s content to expire).


117 posted on 10/02/2012 6:30:30 PM PDT by jurroppi1
[ Post Reply | Private Reply | To 116 | View Replies ]


To: RobertClark

nuts, I should’ve posted the above to you...


118 posted on 10/02/2012 6:31:17 PM PDT by jurroppi1
[ Post Reply | Private Reply | To 117 | View Replies ]

To: jurroppi1
The tags are there now, on the current versions. The original site, the cached versions of which are posted earlier in this thread were 2 to 3 weeks old - I pulled the information using Baidu and various other sources this afternoon.
121 posted on 10/02/2012 7:00:15 PM PDT by RobertClark (Be prepared, be polite, be professional and have a plan to kill everyone you meet.)
[ Post Reply | Private Reply | To 117 | View Replies ]

Free Republic
Browse · Search
News/Activism
Topics · Post Article


FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson