Top search engines

All right, in other parts of this site I've talked a lot about how search engines are a key part of a good website promotion effort and how the use of correct search engine optimization techniques can increase the traffic flow search engines can send to your site. Now, let's take a peek at the engines themselves. This article concentrates on the sizes of different engines and also on their relationships with each other. Reading it should help you decide where to submit your site and where not - some search engines use the same database and submitting the same page several times to one database will just waste your time.

Lets begin with a rough estimate about the number of visitors to the independent engines:

This chart displays the market shares of the twelve biggest independent directories and search engines. The figures are expressed in percentages of the total amount of visitors to different SE's and directories. "Independent" in this case means that the engine has its own database and that you can submit to it. However, it may also use content provided by other search engines to supplement its own results.

Figures quoted in the charts are courtesy of WebSnapshot by MyComputer.com.

When examining the chart (older charts can be found here), keep in mind that the numbers are only estimates. Don't get obsessed about small differences and don't jump to any conclusions. The fact that search engine A might get twice as many visitors as search engine B doesn't mean that you should forget B and only promote your site at A. While the first engine is far more popular, it is also much harder to get a good ranking in it than at engine B. Everything has its ups and downs you know.. Try different things and see what works for you, but try to establish a presence in as many of the top engines as possible.

What has changed since May 2001?

June was the month of paid inclusion programs, as both Altavista and FAST introduced the possibility to pay in exchange for being promptly included in their indices. FAST's program didn't cause much of a stir, but Altavista took some flak for the pricing of its service, which was seen as far too expensive. It is doubtful that AV's program would get many participants in these circumstances, but if FAST manages to handle everything well, its relatively affordable solution might be worth looking at.

Other search engines were also actively improving their services and making deals. Inktomi signed a contract with MSN for the use of Ink's results in MSN Search, which dispelled rumours about Inktomi planning to change its position from a provider of search results and a software vendor to merely selling search software. Google was also busy, introducing an image search engine, new languages for its search interface and a Canadian search engine.

In more unpleasant news, NBCi finally did what was expected and replaced its own search engine with GoTo's results. Good news for those who bid at GoTo, but another loss for everyone who searches the web, as this means that there is one resource less for finding interesting sites.

The charts have seen some changes as well. New engines and directories have appeared on them, with some others dropping out. The new ones are Open Directory (OD), Yahooligans (YL) and Lycos UK (LU). Ask Jeeves, Web.de and surprisingly the Lycos main site lost their positions.

With that being said, let's take a look at where the results shown by different search engines come from:

• YH: Yahoo (learn more about Yahoo, example of a search performed at Yahoo).

  As you can see, it's the leader by far. Yahoo can be searched in two different ways. The first is to browse their directory, the second is to use their search function.

  The search will first display a couple matches from within the Yahoo site under "Inside Yahoo! Matches", if any are available. It will then go through the directory and if anything relevant to the search is found, it is displayed next. Relevant categories are shown first, with relevant sites coming after them. If it doesn't find any matches or has already shown you all the matches from the directory, it will use results from the Google database.

• GG: Google (example of a search performed at Google).

  The provider of secondary results to Yahoo, Google is now one of the most important search engines to be well ranked in. It has been gaining popularity quickly during the last year and there currently seems to be no dark clouds in the horizon.

  Despite its close ties with Yahoo, Google continues to use data from the ODP in the Google Web Directory. This directory is otherwise identical to the ODP directory, but Google uses its own algorithm to arrange the sites instead of the alphabetical system used by ODP. There is a link to the Web Directory on Google's first page, but other than that it doesn't get much exposure.

  When Google is searched, it first gives out a sponsored link, if one is available for the keyword in question. The sponsored listings can be distinguished from the rest by their different-colored background and a small "Sponsored Link" text at the right side of the screen. After that, Google gives a maximum of two related categories from the Google directory, which uses ODP data. Below these there are occasionally a few links to some relevant news. The rest of the results come from Google's own database.

• AV: Altavista (example of a search performed at Altavista).

  The once-proud Altavista has recently been declining and seen its importance reduced. While most still consider it to be a major player, financial difficulties and heavy competition from other search engines have made life hard for AV. Whether it will be able to make a comeback or be doomed to slowly fade away still remains to be seen.

  Like many others, AV has a directory provided by Looksmart on its front page. When Altavista is searched, it starts by showing up to three results under "Featured Sites" if any are available for the keyword used. These are all paid advertisements, some of which are provided by GoTo, others have been sold by Altavista itself.

  Altavista then continues by showing a few relevant news headlines, if they are available and after that uses its own database to show more results. In some searches, there are up to three GoTo results under a "Partner Listings" heading at the bottom of the screen.

• GT: GoTo (example of a search performed at GoTo).

  The biggest of the pay-per-click engines, GoTo gives out relevant sites in its database in the order of "the more the site has paid, the higher it ranks" when searched. If it runs out of paid matches, GoTo draws sites from Inktomi.

  In addition to its search feature, GoTo also has something that resembles a directory on its front page. However, if you take a closer look, you will notice that in reality the categories are just shortcuts to different search phrases.

• EX: Excite (example of a search performed at Excite).

  When searched, Excite gives out matching sites from both its own database and the Looksmart directory. The sites coming from Looksmart use their 'normal' description and title instead of their Looksmart title & description. Most of the top ranking results seem to be coming from Looksmart, so I'd be inclined to say that a Looksmart listing gets pretty much weight in Excite's algorithm. Below these results, at the bottom of the page, three sponsored listings from FindWhat are displayed.

  Excite also has other search modes available. These include 'Photos', 'News' and 'Directory'. The Looksmart directory can also be seen on the front page.

• NB: NBCi (example of a search performed at NBCi).

  All of the search results at NBCi are now provided by GoTo. For this reason, NBCi will be moved out of the independent search engine section in the next update.

• DH: Direct Hit (example of a search performed at Direct Hit).

  Despite being famous for its click popularity technology, Direct Hit has failed to conquer a large enough market share to be able to compete with the big guys. Still, it has managed to sign a few deals with other search engines, causing DH's results to be displayed on Ask Jeeves, Lycos and other SE's as well.

  Direct Hit carries the Open Directory on its front page. It arranges the sites in it with its own click popularity algorithm instead of using ODP's alphabetical sorting method. The search feature has two options, "web sites" (default) and "categories". The default setting first gives three sites from GoTo under "Partner Search Results", then continues by using Direct Hit's own database and displays a maximum of five relevant ODP categories at the bottom of the page.

  The ranking of the sites is determined by counting the clickthroughs done by the searchers. The logic behind this system is that Direct Hit believes people will visit mainly sites that are relevant to what they are searching, thus a site with a large amount of clickthroughs for a particular keyword should be relevant to that word.

  The categories feature on the other hand doesn't use Direct Hit's database at all, instead it displays a maximum of 30 ODP categories relevant to the keyword searched.

• LS: Looksmart (example of a search performed at Looksmart).

  They have their own directory that can be browsed from their front page and also provides content for MSN, AV, Excite and many others. Submitting to the directory costs $199 (Express) or $99 (Basic) for commercial sites. There is a free submission, but it is reserved only for non-profit organisations.

  When Looksmart is searched, it first gives out up to three pages under "Featured Listings", all of which are ads sold by Looksmart. Then come five related categories found in the directory, followed by related sites (also from the directory). After these are exhausted, LS will display additional results from Inktomi.

• OD: Open Directory Project (learn more about ODP, example of a search performed at ODP).

  Don't let the numbers fool you. While the chart would seem to indicate that the influence of ODP is minimal compared to that of for example Altavista, such a view does not have a basis in reality. While the directory itself is not among the most popular sites in the world, ODP's data is being used by many search engines, which makes submitting to it a very smart move.

  Because it is a directory, the ODP of course carries its own directory on its front page. If the search feature is used, ODP first displays up to five relevant categories and shows matching sites after them.

• YL: Yahooligans (example of a search executed at Yahooligans).

  Yahooligans is Yahoo's directory for kids. Its structure resembles that of Yahoo, but the content is completely different.

  The Yahooligans directory is naturally present on the front page and can be browsed, but Yahooligans also has a search feature. The search first gives a couple matches from the Yahooligans site under the heading "Inside Yahooligans! Matches", if any are available. It then lists any matching categories and displays matching sites after that. Unlike Yahoo, Yahooligans does not use Google as a secondary results provider.

• LU: Lycos UK (example of a search performed at Lycos UK).

  Lycos UK is the British version of the well-known Lycos search engine. It offers three different search modes, UK (default), Worldwide (Uses the standard Lycos index) and Shopping.

  The UK search mode gives first up to three relevant categories from the Lycos UK directory and then shows pages considered to be especially relevant for people who live in UK from the Lycos index of sites. Nearly all of the pages that appear in this search are located in domains that end in ".uk".

• FI: Fireball (example of a search performed at Fireball).

  Another German search engine that also features its own directory on the front page. The search feature can be used to search in several languages, the default being deutsch (German). This mode searches the Fireball database for matches, showing relevant sites first and relevant categories at the bottom of the page. Searches in other languages use Altavista's language specific search features. You can submit your site to Fireball, provided that it is in German.

These things give us a few ideas. First, the Yahoo, Looksmart and ODP (in that order) directories are very important. Second, Google is arguably the most important search engine, serving a huge number of searchers on its own and as the provider of secondary results to Yahoo, so being well placed in it is very important at the moment.

Third, Inktomi still provides search results for a massive amount of engines, but they seem to be losing ground at an accelerating pace. While they shouldn't be entirely forgotten, I wouldn't recommend betting too much on Inktomi in the current situation. However, should you wish to seriously attempt to receive some traffic from Inktomi, remember that they mostly provide secondary results that are only seen by users if the keyword used is a rare one. This could make it more profitable to try to rank high for more obscure words in Ink than in other engines.

Now, what you read above was what I consider to be the important stuff. If you're feeling that you've had quite enough information about the popularity of different engines already, don't feel guilty about not reading the stuff below. Why? Because next we're going to take a look at non-independent engines, that means engines that don't have their own database. These engines have no individual content and cannot be submitted to, so you shouldn't worry too much about them. But if you have the desire to know what databases are used most, well, you're going to have to continue reading..

Looks like you've made up your mind, so here's a chart of visitors to non-independent engines:

As you can see from the chart, non-independent engines get massive traffic too. But remember that searching is usually not the primary feature of these sites. An independent engine probably gets searched more than a non-independent one even if the number of visitors is the same.

As with the previous chart, take these figures with a grain of salt. These figures can include visits to parts of the site that do not have a search feature, ie. MSN's figures may include visitors to their news service and so on. Anyway, lets proceed and look at who uses who's database:

• MS: MSN (example of a search performed at MSN).

  Bill's portal has become one of the biggest players in the search engine industry. Their search function will first give out a few "featured sites", if any are available for the keyword searched. These are mostly paid advertisements and Microsoft's own sites. After that, the user is shown a "Get the Top <number, ten at most> Most Popular Sites for <keyword>" link. If it is clicked, MSN will display some Direct Hit results, which are determined by user selection traffic.

  Moving on, it will browse through Looksmart's directory, showing any relevant sites in the directory. When every match found there is shown, it will use the Inktomi database.

  Note that if the keyword searched is of adult nature, there is a small exception to what is described above. Before showing any results, the user is asked whether he wishes to use a certain adult search engine or to see normal MSN results like explained above.

• AO: AOL (example of a search performed at AOL).

  A big ISP and a big site, providing a nice amount of traffic to those who rank high on its search results. When it is searched, it begins by showing a few sites under 'Recommended Sites', if any are available for the keyword. These consist of AOL's own sites and paid advertisements. AOL then continues by giving out three 'Sponsored links', which come from GoTo.

  After those have been shown, it gives out matching sites from the ODP directory, but ranks them differently than the ODP itself. It seems that AOL spiders and indexes ODP sites, unlike ODP, which uses only the title, URL and description provided to it. This means that if your ODP description/title/URL doesn't contain the word "promotion", you won't be found on ODP when that word is searched. But if one or more of your site's pages contain that word, AOL will see you as relevant even when ODP doesn't!

  Up to five relevant ODP categories are also shown at the bottom of the results page. Inktomi provides the secondary results, but relevant sites found in its database are only shown if the ODP data doesn't produce even a single match, which is quite rare.

• NS: Netscape (example of a search performed at Netscape).

  When searched in the default mode, "Search the Web", Netscape first gives out two results from GoTo under "Partner Search results". If the keyword you entered is the name of a major company (ie. Netscape), the official site of this company will then be shown under "Official Web Sites".

  The search then gives up to ten matches from within the Netscape site and goes on to give up to 10 related ODP categories. After those it shows relevant sites from the ODP directory. If no matching ODP sites are found, it uses Google, but it does not show Google results if there is even one matching ODP site.

  Netscape can also be told to search from Ask Jeeves, Google, GoTo, HotBot, Looksmart, Altavista or Lycos databases. However, I believe that an overwhelming majority at least begins their search by using the default mode.

• DP: Dogpile (example of a search performed at Dogpile).

  Dogpile has the Looksmart directory on its front page. The search feature offers multiple options, which include "The Web" (default), "Images", "Audio/MP3", "Auctions", "Files", "News", "FTP", "Discussion" and "Multimedia". In "The Web" mode, Dogpile draws its data from multiple search engines and directories. Sponsored results provided by GoTo, FindWhat and About.com's Sprinks to mention a few are prominently displayed.

• CN: CNet's Search.com (example of a search performed at Search.com).

  This metasearch engine offers a directory of other search providers on its front page. The directory lists various topic specific search engines which users may conduct searches in.

  Search.com's main search feature draws its results from several search engines. It begins by giving out five paid results from GoTo under Search Partners, if these are available for the keyword in question. Next, the user is offered up to six matches from various CNET properties under "Computing". The rest of the results are under "Web Pages" and are provided by Lycos, The Open Directory, Sprinks, FindWhat, Direct Hit and various other search engines and directories.

• HB: HotBot (example of a search performed at HotBot).

  HotBot has the Open Directory Project-directory on its front page. It is slightly modified from the original version by adding up to five related results from GoTo under "Featured Listings" at the top of each category. For example, the "Games" section of the directory displays the five top ranking sites from GoTo for the keyword "Games".

  In its search results, HB first displays up to three results from GoTo, if these are available. Occasionally these are placed below the Lycos Network results under "Products & Services" instead of showing them at the top.

  The GoTo results, if located at the top like usual, are followed by a few links under the heading "Search Partners", all of which are paid ads. Next come results from the Lycos Network, which consist of different parts of the Lycos site. Then Lycos shows the 10 best matching Direct Hit results, if they are available. If there are less than 10 Direct Hit results available, HotBot displays a link that says "Get the Top <number of available Direct Hit results> sites for 'keyword'" instead. One more "Featured Site" is listed after the 10th result. It comes from GoTo, just like the "Featured Listings" at the directory.

  After those results have been shown, HB gives a maximum of five related ODP categories at the bottom of the page and proceeds to give out sites from both the ODP site database and Inktomi (you can submit to Inktomi by submitting to Anzwers). You can tell sites coming from ODP by looking at their listing; all ODP sites will have a line starting with "More like this:" followed by the ODP category they are listed in. Sites coming from Inktomi or Direct Hit will have a "See results from this site only"-line. The sites coming from ODP will use their ODP description and Title, while the sites coming from Inktomi will naturally have their title and description taken from the page itself. Keep in mind that the HotBot version of the ODP directory is pretty out-dated, so it has some differences when compared with the most current ODP version.

• MC: Metacrawler (example of a search performed at Metacrawler).

  When Metacrawler is searched with the default settings, it displays up to three sites from GoTo under "Featured Search Results". After them, it shows results from various search engines, for example Altavista, Direct Hit, GoTo, Kanoodle, Looksmart, FindWhat and About.com's Sprinks. The matches provided by these search engines do not seem to be arranged into any specific order.

  Metacrawler also has other search modes in addition to the default called "The Web". These include "Directory", "Audio/MP3", "Files", "Images", "Newsgroups", "Auctions" and "Multimedia".

• MA: Mamma (example of a search performed at Mamma).

  The results at Mamma begin with a few sites under the heading "Mamma Recommends..", which are of course paid advertisements. After those have been shown, Mamma displays matches from the databases of other search engines and directories. These include Yahoo, Ask Jeeves, FindWhat and several others.

  Mamma also offers the possibility to search for news, MP3's, Audio, Images or Video.