The Internet Archive discovers and captures web pages through many different web crawls.
At any given time several distinct crawls are running, some for months, and some every day or longer.
View the web archive through the Wayback Machine.
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What?s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa?s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available ?warts and all? for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you?re hoping to do with it. We may not be able to say ?yes? to all requests, since we?re just figuring out whether this is a good idea, but everyone will be considered.
I-90 Bridge Lawsuit Moves¬†To Kittitas County Superior Court
On the heels of a recent ruling by the Washington State Supreme Court, a legal fight to stop the seven mile center express roadway of Interstate-90, including the Lake Washington Floating Bridge from being transferred illegally has moved to a new battleground – Kittitas County Superior Court in Ellensburg.
Speaking about choosing the Kittitas County Superior Court to place the suit, George Kargianis and Phil Talmadge, counsel for the plaintiffs, said,¬†‚ÄúThis is not a Puget Sound only issue.¬†¬†What happens in that seven mile stretch and over the bridge affects everyone in the state.‚ÄĚ Counsel explained that Interstate-90 serves as the gateway to Snoqualmie Pass and Puget Sound for all of Washington‚Äôs agriculture and commerce.¬†¬†‚ÄúWe are asking Kittitas County Superior Court to hear our case because the County is the most central in the state and borders Interstate I-90 and linking eastern and western Washington.‚ÄĚ
The lawsuit first came about nearly two years ago when the Washington State Department of Transportation began preparations to transfer use of the I-90 center express lanes to Sound Transit for the purpose of building light rail across the bridge, over Mercer Island and on through the seven mile corridor.
But a group of leaders from around the state led by Bellevue businessman Kemper Freeman filed a Writ of Prohibition in the Supreme Court to stop the state from performing an illegal act.¬†¬†A Writ of Prohibition is a special emergency legal action that bypasses lower courts when a legal action by the state is occurring or about to occur and immediate determination is needed.
The group contends the 18th Amendment to the Washington State Constitution prohibits state highways that are built using gas tax revenues, and that are not surplus roads, from being converted to ‚Äúnon-highway purposes,‚ÄĚ as defined in the law.¬†¬†The State Supreme Court has previously ruled that rail is not a highway purpose.
‚ÄúThose seven miles are not surplus by anyone‚Äôs estimation.¬†¬†And light rail is not a highway purpose,‚ÄĚ stated Freeman at a recent meeting of the plaintiffs.
In the late April ruling, the High Court in essence decided that the case was not ready to be heard by the Supreme Court and informed the plaintiffs that arguments should be taken up in superior court first.
Former State Senator Jim Horn of Mercer Island, another plaintiff and past Chair of the State Senate Transportation Committee, says that losing the center express lanes reduces the roadway capacity essentially one third by altering the roadway from ten lanes to eight lanes.
‚ÄúThe I-90 center-roadway is too valuable, and our highway needs are far too great to have it converted to single purpose non-highway use by light rail.¬†¬†Doing so would certainly raise the misery index for those stuck in traffic day in and day out,‚ÄĚ says Horn.
Bryan Boehm of American Container Transport in Ellensburg, one of the eastern Washington plaintiffs in the lawsuit, says consumers could see higher prices in the store, and that exports could be hurt because of increased delay times for trucks coming over Snoqualmie Pass.¬†¬†¬†Says Boehm,¬†‚ÄúThousands of trucks from Boston to Seattle use I-90 and the floating bridge corridor every day, and we are very concerned about getting bottled up in the last seven miles.‚ÄĚ
Another of the plaintiffs, Sarah Rindlaub, Board Member with the Washington Policy Center in Seattle is also concerned about the consequences of such a highway transfer.¬†¬†‚ÄúViolating the law is never an option.¬†¬†The 18th Amendment came about in the first place to protect the people‚Äôs gas taxes from being hijacked for other causes, no matter what the purpose.‚ÄĚ
The timetable for a Kittitas County Superior Court ruling could be several months, at which time, other appeals are possible. According to legal counsel, in time, the case will likely find itself right back on the doorstep of the High Court.
Expressing his personal reasons for continuing the legal action, says Kemper Freeman:¬†‚ÄúIt‚Äôs because there‚Äôs never a right time to do a wrong thing.¬†¬†And doing the wrong thing in this case will hurt everyone in the state for years to come.‚ÄĚ