The Next Generation of Search
New search engines are taking on the old guard, armed with radical new technology
FROM WEDNESDAY, MAY 5, 1999
The truth is out there -- the real question is, where the hell is it in all this crap? Search has always been an integral part of how people use the Web, since its earliest days, and the Web's first big brands -- Yahoo, Excite, Lycos -- were all search engines. But can they still cut the mustard? A new generation of search services is springing up, with names like Google and FAST, armed with next-generation technology, and they say they have the power to supplant their elders and finally make sense of the Web.
The funny thing about the Web as a medium is that it's always been plagued by too much content, rather than too little. A now-infamous report issued last summer by the NEC Research Institute announced that for all their inflated stock prices, the major search engines were covering at best only a fraction of the Web. Forrester Research estimates the size of the Web at 500 to 600 million pages; AltaVista, which claims to index the most pages of any major search engine, covers only 150 million. The best bet, the report stated, may be search engines like Metacrawler, which simply aggregate search results from other engines.
Enter Fast Search & Transfer (FAST),
an Oslo-based company founded in 1997 by a group of Norwegian academics. It
claims that its new FAST Search service, launched yesterday, will be bigger, speedier and more accurate than the
existing major search engines. The company is vague about the technical
details behind its new search service, but it hints that its solution lies in the scalability of
FAST's software -- its ability to handle ever-increasing amounts of data --
and the distributed structure of its hardware -– i.e., it runs on lots of small,
inexpensive machines, rather than a few big ones, so that adding extra computing power
is cheap and easy.
All the more so because FAST has found a powerful business ally in Dell Computer Corporation. Like Compaq, which owns
AltaVista, Dell seems to have discovered that a search engine makes a stylish
accessory for a computer manufacturer looking to expand into web servers and
web retailing. Can FAST put its search muscle where its mouth is? Already it
claims to cover an impressive 80 million pages -- more than Excite, Lycos or Infoseek,
according to Search Engine Watch -- and the company says it is on track for 200 million pages by this summer.
But size isn't everything, and search results are useless if they
aren't accurate. Consider the case of Google, a search engine developed by Stanford grad students. It indexes only about 60 million pages, but it does so extremely intelligently. Google rates the relevance of a web page to a particular search query based in part
on how many other web pages link to that page. Clever, eh? By surveying more than a billion links Web-wide, Google essentially polls the rest of the Internet and asks it how important it thinks a given web page is. As a result, Google is amazingly good at giving search results that relate to what you're looking for.
Google began in the ivory tower, but it's now a private company, and its web site states that it is "unable to speculate" as to whether or not it will go public -- a sure sign that an IPO is in the air. For now, it's a smart solution to a thorny problem that too many companies try to solve with snappy TV commercials and a hip mascot. Let's hope Google's market cap won't go to its
WRITE TO US