The Wayback Machine - https://web.archive.org/all/20040615105352/http://www.robotstxt.org:80/wc/robots.html

The Web Robots Pages

The Web Robots Pages

The Web Robots Pages

Web Robots are programs that traverse the Web automatically. Some people call them Web Wanderers, Crawlers, or Spiders. These pages have further information about these Web Robots.

The Web Robots FAQ
Frequently Asked Questions about Web Robots, from Web users, Web authors, and Robot implementors.

Robots Exclusion
Find out what you can do to direct robots that visit your Web site.

A List of Robots
A database of currently known robots, with descriptions and contact details.

The Robots Mailing List
An archived mailing list for discussion of technical aspects of designing, building, and operating Web Robots.

Articles and Papers
Background reading for people interested in Web Robots

Related Sites
Some references to other sites that concern Web Robots.

May	JUN	Jul
	15
2003	2004	2005

The Web Robots FAQ	Frequently Asked Questions about Web Robots, from Web users, Web authors, and Robot implementors.
Robots Exclusion	Find out what you can do to direct robots that visit your Web site.
A List of Robots	A database of currently known robots, with descriptions and contact details.
The Robots Mailing List	An archived mailing list for discussion of technical aspects of designing, building, and operating Web Robots.
Articles and Papers	Background reading for people interested in Web Robots
Related Sites	Some references to other sites that concern Web Robots.