COLLECTED BY
Organization:
Alexa Crawls
Starting in 1996,
Alexa Internet has been donating their crawl data to the Internet Archive. Flowing in every day, these data are added to the
Wayback Machine after an embargo period.
Crawl DX from Alexa Internet. This data is currently not publicly accessible.
The Wayback Machine - https://web.archive.org/all/20040610232833/http://www.robotstxt.org:80/wc/meta-user.html
HTML Author's Guide to the Robots META tag
The Web Robots Pages
HTML Author's Guide
to the Robots META tag.
The Robots META tag is a simple mechanism to indicate to visiting
Web Robots if a page should be indexed, or links on the page should be
followed.
It differs from the Protocol
for Robots Exclusion in that you need no effort or permission from
your Web Server Administrator.
Note: Currently only few robots support this tag!
Where to put the Robots META tag
Like any META tag it should be placed in the HEAD section of
an HTML page:
<html>
<head>
<meta name="robots" content="noindex,nofollow">
<meta name="description" content="This page ....">
<title>...</title>
</head>
<body>
...
What to put into the Robots META tag
The content of the Robots META tag contains directives separated by
commas. The currently defined directives are [NO]INDEX and
[NO]FOLLOW. The INDEX directive specifies if an indexing robot should
index the page. The FOLLOW directive specifies if a robot is to follow
links on the page. The defaults are INDEX and FOLLOW. The values ALL
and NONE set all directives on or off:
ALL=INDEX,FOLLOW and NONE=NOINDEX,NOFOLLOW.
Some examples:
<meta name="robots" content="index,follow">
<meta name="robots" content="noindex,follow">
<meta name="robots" content="index,nofollow">
<meta name="robots" content="noindex,nofollow">
Note the "robots" name of the tag and the content are case insensitive.
You obviously should not specify conflicting or
repeating directives such as:
<meta name="robots" content="INDEX,NOINDEX,NOFOLLOW,FOLLOW,FOLLOW">
A formal syntax for the Robots META tag content is:
content = all | none | directives
all = "ALL"
none = "NONE"
directives = directive ["," directives]
directive = index | follow
index = "INDEX" | "NOINDEX"
follow = "FOLLOW" | "NOFOLLOW"