http://obamareleaseyourrecords.blogspot.com/2011/10/new-york-state-board-of-elections.html
New York State Board of Elections Website Blocking Access To Natural Born Citizen Requirements
Indeed, exactly the same thing as I discussed up-thread (my first response to Danae regarding code snippets). That is simply telling the crawler/bot not to index the page. That code isn’t blocking access to anything, it is just asking the Google bot specifically not to index the information (first line), and asking all other bots to please do the same on the second line.
The generally accepted method of doing that is the robots.txt file. Regardless of method, some bots will ignore the request.
Dumping info and keeping the same info from being archived by (kind) archiving bots is especially sneaky under these circumstances though.
That’s interesting.
We are dealing with a determined and well-trained adversary. We must act accordingly.
>