Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them.
http://www.robotstxt.org/robotstxt.html
Oh okay; thanks.
Found that article on other sites, and here is one: