Starting in 1996, Alexa Internet has been donating their crawl data to the Internet Archive. Flowing in every day, these data are added to the Wayback Machine after an embargo period.

Collection: 51_crawl

this data is currently not publicly accessible.

Google Sitemaps explained

Ultimately all the major search engines want great content. But sending a small program (robot or spider) out to every page to see if it has been updated is inefficient at best. That’s why Google has created “Google Sitemaps”.

What are the big advantages to Google’s Sitemaps?

1) You have control over a few important variables.

• Location – Tell Google where your web pages are located (i.e. each page’s URL).
• Priority – Inform Google as to the relative importance of all pages on your website.
• Last modified – Tell Google when you last changed a page. This saves the robots a great deal of time (they don’t have to re-crawl pages that have not changed).
• Change Frequency – Tell Google how often you update a page.

2) You get to find out how Google sees your site.

• Crawl errors – Google will tell you which pages they had trouble reading.
• Statistics – Google will create reports of your top search queries and the terms that sent you the most traffic.
• Content Density – Google will create a report outlining how it sees your content: keyword density on the page and in the links pointing to your website.
• Index information – A Google report outlining which of your pages are in the index and when the last time Googlebot crawled your website.
• Violations of Google’s webmaster guidelines – If Google finds a blatant SEO mistake, they may tell you. Once fixed, you can request re-inclusion.
• Robots.txt – Google will report on whether or not your robots.txt file is set up properly.

3) It is fairly straightforward and it’s FREE.

• You need to create a Google account.
• You need to sign into Google Sitemaps using your Google account.
• You must prove you own the site by uploading a file via ftp.

To submit a Sitemap for a site:

1. Create a Sitemap in a supported format.
2. Add your site to your Google Sitemaps account.
3. Add the Sitemap to your Google Sitemaps account.
4. Update your Sitemap when your site changes.

Google Sitemaps Risks and Comments

We believe Google sitemaps should be implemented. However, it is important to understand the risks:

1. Many business owners don’t view Google as another corporate entity; they see Google as a friend who helps bring traffic to their website. Everybody should be concerned with the amount of information Google is collecting, and whether they are giving out this information too freely.
2. The more Google knows about you and your business, the more money they can charge for each sale/lead they provide you with. Google Sitemaps is a great tool that can help you understand your organic rankings, but Google is using the information to define their Pay Per Click strategy and your Minimum PPC bid price.
3. If you have the Google toolbar installed on your computer, Google already knows everywhere you go and everything you look at. Google sitemaps allows them to see everything your customers do when they are on your website.

Google Sitemap Tool – Create your feed

VKI Studios recommends - http://johannesmueller.com/gs/

The GSiteCrawler is an excellent tool that crawls a website and checks Google to create a list of pages within the website. Once the crawl is complete, you can change the priority and frequency of each page crawled and use this data to generate a Google Sitemap.