Welcome to www.robotstxt.org, the new home of The Web Robots Pages.
This is the main source for information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.
| Welcome to www.robotstxt.org, the new home of The Web Robots Pages. This is the main source for information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots. |
Thanks to WebCrawler and Excite for having hosted these pages at http://info.webcrawler.com/mak/projects/robots/robots.html over the years. || Thanks also to Diesel Sweeties for the Clango cartoon. || Maintainer: Martijn Koster