Internet; the beginning
The first search engines
The first modern search engines.
It all started in the late 60's. The world was threatened by the cold war. The defend department of the US was looking for a way to protect their communication lines. In that time when one part of their network would be hit by a Russian bomb, the whole communication line would be down. With this in mind the Americans began with the development of Arpanet, the forerunner of the present internet. The advance of Arpanet was that the communication lines between multiple computers varied. So there was no single line but there were multiple ones.
In the 80's universities saw the great advances of such a network. So they created their own. In the years that came many universities and other institutions connected to the net. They used the same language: TCP/IP. The internet was born.
When the internet was growing the first search engines appeared. Archie, derive from "Archive", is called to be the grandfather of all search engines. It was developed in 1990 by Alan Emtage. This was the beginning of the history of search engines. In that time there was no World Wide Web. Files were shared over the network by using the FTP-protocol. By then sharing files happened at the following way: A person decides he wants to make files available. He makes sure he has installed a FTP-server on his computer. When someone else on the internet wants to get files from this computer, he or she connects to it through another program called an FTP client. When the client has connected to the IP-address of the server he can retrieve files. By that time the problem was that the availability of a file had to be announced by mouth. (through mailing lists or discussions forums)
Archie changed all that. Archie's gatherer scoured FTP sites across the Internet and indexed all of the files it found. Now, there was a possibility for users to find files matching there search queries. The first search engine was born...........
A few years later (1993) an other search engine took a look: " Veronica ".
Veronica worked similar to Archie but there was one difference. While Archie indexed all anonymous FTP-sites, Veronica indexed all gopher documents around the world. As the same as Archie, by typing a query in the searchbox you could find the information you need.
Gopher (1991) is the forerunner of the World Wide Web. By browsing through menus on a particular server, you could retrieve the information you need. The difference between each other is that the last one has a more user friendly interface and it offers much more graphical en multimedia possibilities.
By this time, along with the appearance of the WWW, the first robot appeared. It was called " the lone wanderer". The main purpose of this robot was to check the web growth. At the beginning it only counted the webservers. Later it captured URL's. The database was called "the Wandex"
In October 1993, short after the introduction of "the lone wanderer", ALIWEB made his debut. It wasn't a big success. ALIWEB was not a spider who indexes webpages automatically. The webmaster had to create his own index and then send it to ALIWEB. Because few people had the knowledge to do that, ALIWEB wasn't quite a success.
The first three (automatically) spiders were founded by the end of 1993 and begin 1994. They were respectively called: JumpStation, the World Wide Web Worm, and the Repository-Based Software Engineering (RBSE) spider. These spiders jumped from link to link and indexed the pages they met. But they weren't as advanced as the search engines of these days. They only gathered information about headers, titles and URLS of WebPages. The search interfaces weren't also as advanced as nowadays. Another weak point was that the order of the results were completely irrelevant. Only the RSBE spider had some sort of relevance ranking system.
When the web became larger, these spiders became slower. There was a need for an other kind (faster) of spider.
Fist full-text spider: WebCrawler
In 1994 students of the university of Washington created the first spider which makes documents full text searchable. They gave it the name "WebCrawler". Short after it's introduction this search engine became very popular. That's why in a short time it had to deal with bandwidth troubles. AOL saved WebCrawler. And in 1997 Excite bought out WebCrawler from AOL. In the first years WebCrawler was an example for many starting search engines.
In July 1994 Lycos was starting to serve the information minded people. By that time it was one of the most used search engines. In 1995 they had indexed an estimated 90 percent of the web. (more then 10 million documents) Lycos didn't
index documents full-text. They indexed only the first 20 sentences of a document, the first 200 words of the header, and a group of 100 words that were the most relevant. Lycos leading position can be explained by the large index, the great relevancy and the quick response. Unfortunately by that time there weren't many advanced search options, such as searching on document-type, searching documents in multiple languages, etc.
Excite is called to be one of the first modern search engines. It went online in December 1995. By that time Excite became an often used and popular search engine. It was a search engine full of life. The followings businesses happened:
- 1996: Excite acquired Magellan and WebCrawler
- 1996: Excite includes its directory service, Excite Channels
Excite used a search algorithm that index full-text the main page of a website. Also they used a rank system to determine which page was the most relevant.
In mid 1999 Excite lost his leading position on the search engine market. In 2002 the search engine functionality has been removed. Now the search results comes from different search engines ( Google, Overture, Inktomi), and directory results comes from Open Directory. One thing that didn't change is that Excite is still a portal, just as when they started.
First metasearchengine: MetaCrawler
Because different search engines turned up with different results for the same query , it was in general not easy to find exactly what you wanted. So people were looking for a way to use different search engines simultaneously. The solution was developed in 1995 by Eric Selburg: MetaCrawler. This search engine searched Lycos, AltaVista, Yahoo, Excite, WebCrawler and Infoseek at the same time. A disadvantage was that the response time wasn't as fast as people wanted it to be.
By this time AltaVista was one of the major search engines. It was the Google of those days. AltaVista made a great step forward with his large scale of advanced search options. It was the first to use natural language queries. It was also the first to implement advanced search options, such as the use of Boolean search queries. People could search for text in images names, titles, java applets etc. Furthermore, users could retrieve newsgroup articles. AltaVista was by that time a very innovative, most used search engine. But unfortunately in 1999/2000 the success came to an end because an future giant raise up..............Google.
Northern Light was founded in August 1997. And was by that time the biggest search engine. These search engine was very popular among professionals. By the general public the popularity wasn't as big as it could be. Besides the large database, also the search technology was very well build. They made use of so called clusters; when you were looking for a particular subject, the search engine made automatically clusters to refine your search. Unfortunately in 2002 Northern Light had to stop their services. His famous collection of magazine articles (30 million) can still be seen against an amount of money. They are now specialized in selling their search technology.
Google (= Googol; 1 followed by 100 zeros)
In 1998 Google was created by two Stanford Ph.D. students, Larry Page and Sergey Brin. When they invented their technology, nobody was interested. Later on, David Filo from Yahoo! said their technology was solid. He encouraged them to start a company themselves. They decided to follow his advice. At the beginning Sergey and Larry had some troubles to find some investors. But in the end they gathered $1000.000, what was quite an amount of money. As soon as Google opened his doors, the search engine became each month more popular. At the end of 1998 it was one of the hundred best visited websites. As we probably all know, nowadays Google is known as the best search engine. Especially the user-friendly interface and the high level of relevant results are people very pleased with.
There are some newcomers with potential who can threaten Google in the future.
A few examples are Teoma and Wisenut. Teoma has a very good relevancy technique, but their database is small. Wisenut has a large database and a pretty good relevance ranking technique. From the search engine history till now, the developments have followed up each other in a fast way. We will see what more will happen in the future.