Top search engines

All right, in other parts of this site I've talked a lot about how search engines are useful in driving traffic to your site and how to improve and increase that traffic flow. Now, let's take a peek at the engines themselves. This section concentrates on the sizes of different engines and also on their relationships with each other. Reading this should help you decide where to submit your site and where not - some search engines use the same database and submitting the same page several times to one database will just waste your time.

Lets begin with a rough estimate about the number of visitors to the independent engines:

This chart shows the number of visitors in the ten biggest independent engines and directories. "Independent" in this case means that the engine has it's own database and that you can submit to it. However, it may also use content from other providers databases in addition to it's own.

Figures quoted in the charts are courtesy of PCData.

When examining the chart (older charts can be found here), keep in mind that the numbers are only estimates. Don't get obsessed about small differences in visitors and don't jump to any conclusions. The fact that Yahoo gets almost three times more visitors than Altavista doesn't mean that you should forget Altavista and only promote in Yahoo. It's true that Yahoo is far more popular, but it's also much harder to get a good ranking in Yahoo than in Altavista. Everything has its ups and downs you know.. Try different things and see what works for you, but try to establish a presence in as many of the top engines as possible.

What has changed since September 2000?

Like I anticipated in the previous report, I had to change the scale of the independent engines chart. The scale is now 0 to 70 million, while in the previous charts it was 0 to 60 million. This was prompted by the continuing growth of Yahoo, which finally broke the figure of 60 million visitors. I also did some other minor modifications, mostly to make room for a new engine in the chart. The newcomer is Google, which has been growing in importance for the past few months. It will carry the symbol 'GG' and its color will be black.

In other news, NBCi seems to be doing pretty well, since it has increased its visitors from last month's 17.6 million to 21.3 million. GoTo and Looksmart seem to be gaining new partners, like they did last month. This time Infospace changed its directory provider to Looksmart from ODP, and AOL started using GoTo results in its search engine. The Open Directory Project seems to continuously lose support, as Looksmart is currently paying other sites to use its directory. I guess the only thing that can beat a freebie is cash... Apart from these, this month hasn't shown any drastic changes in one direction or another. That's it for now, hopefully you'll catch me next month for another update.

PS. I'm currently creating these charts with Photoshop, which is very time-consuming work. If you happen to know of a free program for Windows that creates decent-looking charts, please give me a heads up! It would enable me to post new figures faster than now.

Let's take a peek at what's behind those numbers and names:

• YH: Yahoo (learn more about Yahoo). As you can see, it's the leader by far. Yahoo can be searched in two different ways. The first is to browse their directory, the second is to use their search function. The search will first go through the directory and if it finds any matches, it displays them at the top of the list. Relevant categories are shown first, with relevant sites coming after them. If it doesn't find any matches or has already shown you all the matches from the directory, it will use results from the Google database.

• LY: Lycos. They have acquired the HotBot engine but haven't completely combined it with their own. HotBot has the Open Directory Project-directory (learn more about ODP) on their front page. In its search results, HB first displays the 10 best matching Direct Hit results, if they are available. If there are less than 10 Direct Hit results available, HotBot displays a link that says "Get the Top <number of available Direct Hit results> sites for 'keyword'" instead. The ranking of the Direct Hit results is determined by counting the clickthroughs done by the searchers. The logic is that people will visit mainly sites that are relevant to what they are searching, thus a site with a large amount of clickthroughs for a particular keyword should be relevant to that word. After those results have been shown, HB gives a maximum of five related ODP categories at the bottom of the page and proceeds to give out results from both Inktomi (you can submit to Inktomi by submitting to Anzwers) and the ODP site database. You can tell sites coming from ODP by looking at their listing; all ODP sites will have a line starting with "More like this:" followed by the ODP category they are listed in. Sites coming from Inktomi will have a "See results from this site only"-line. The sites coming from ODP will use their ODP description and Title, while the sites coming from Inktomi will naturally have their title and description taken from the page itself.

  The actual Lycos engine also carries some parts of the ODP directory at the bottom of its front page. When searched, Lycos first gives a few 'popular sites', which consist of relevant sections of the Lycos site if any are available. If there are insufficent on-topic results within the Lycos site, the popular sites are taken from Direct Hit's results. You can distinguish between these by looking at the line below 'Popular' heading; if the results are from the Lycos site, it will read "<amount> of the Web sites reviewed by Lycos Editors match your search", and if they are from Direct Hit, the line will read "<amount> Web sites were selected based on user selection traffic". Lycos then proceeds by displaying up to four relevant ODP categories and uses FAST for the rest of the results.

• NB: NBCi (former Snap). The upper part of their front page contains their very own directory for users to browse. If NBCi is searched, it begins by giving out a few sponsored listings, if any are available. They then give out relevant sites from the NBCi Directory and show up to seven matching categories at the bottom of the page. When the NBCi Directory runs out of matches, the NBCi LiveDirectory is used. The LiveDirectory is a set of user-submitted sites that have not been reviewed by the NBCi editors and promoted to the Directory. If a site proves to be popular among users when it is in the LiveDirectory, it will get reviewed and if it is good enough, it gets promoted to the Directory. When the LiveDirectory runs out of matches, Inktomi is used for the rest of the results.

• EX: Excite. When searched, Excite gives out matching sites from both its own database and the Looksmart directory. The sites coming from Looksmart use their 'normal' description and title instead of their Looksmart title & description. Excite also carries the Looksmart directory on its front page.

• AV: Altavista. Like many others, AV has a directory provided by Looksmart on its front page. When Altavista is searched, it displays related Looksmart categories (up to six), if any are found, at the top of the page. Altavista then continues by giving out sites from its own database.

• AS: Ask Jeeves. This engine is designed to answer questions asked by the user instead of just searching matches in their database for words or phrases. Human editors review all submissions, so unlike most search engines AS does not accept nearly every site. When Ask Jeeves is searched, it first shows any specific answers it might have in its database. It then shows a maximum of five matching sites provided by Direct Hit. After that, they preform a search in About.com's Sprinks, 4Anything Network, StreamSearch, Mamma and GoTo for the question asked and allow the user to visit any of the top ten matching sites in each of these engines if he so desires.

• AB: About.com. When searched, it begins by displaying a maximum of 10 relevant pages in the About.com site. It then continues by showing up to ten PPC (pay per click) results from their PPC program called Sprinks, provided that someone has bid for that keyword. The program is very similar to what GoTo has been offering for a long time. You can spot a Sprinks listing by looking at the end of the site's description. If it ends with "Listing fee <amount>", the site is from Sprinks. The AB site itself is divided into categories and each category has it's own editor. You can submit to it by E-mailing the editor of a category related to your site, but most editors are very picky and only the best sites get in.

• LS: Looksmart. They have their own directory that can be browsed from their front page and also provides content for MSN, AV, Excite and many others. Submitting to the directory costs $199 (Express) or $79 (Basic) for commercial sites. There is a free submission, but it is reserved only for non-profit organisations. When Looksmart is searched, it first gives out any related categories found in the directory, then continues by giving out related sites (also from the directory) and uses Inktomi for the rest of the results.

• GO: Go.com. This Disney-owned portal has swallowed up Infoseek, but seems to have lost some of its visitors and is now claimed to be focusing more on entertainment than typical portal features. In their search results, they give related categories from their directory first, then continue by giving out five related sites from their directory and five related sites from their Infoseek search engine database. The user may then select to either view more site results from the directory or Infoseek, provided that any are available. Although they use their directory in searches, it is not displayed on their front page.

• GT: GoTo. The biggest (and best according to many people) of the pay-per-click engines, its gives out relevant sites in it's database in the order of "who pays most, gets the first place" when searched. If it runs out of matches in it's own database, it draws sites from Inktomi. GoTo also has a directory including only paying sites on its front page.

• GG: Google. This former small-time engine has been rising in importance, and after capturing the position of Yahoo's secondary results provider from Inktomi, it can be seen as one of the big ones. When Google is searched, it first gives out a sponsored link, if one is available for the keyword in question. The sponsored listings can be distinguished from the rest by a different-colored background and a small "Sponsored Link" text at the right side of the screen. After that, Google gives a maximum of two related categories from the Google directory, which uses ODP data. The rest of the results come from Google's own database.

These things give us a few ideas. First, the Yahoo, Looksmart and ODP (in that order) directories are very important. The Go and Snap directories are also useful, but traffic from them is slow compared to the "big three". Second, despite getting dumped by Yahoo, Inktomi is still provides search result for a massive amount of engines. That is something to remember, but keep in mind that Inktomi is usually not the first source of results; even if you're no. 1 in Inktomi, you might not show up on some sites that use the Inktomi database. Still, the database gets used a lot and you should try to use this fact to your benefit, especially if your keywords are uncommon/very specific. With rarely used keywords, the sites using Inktomi usually need to draw results from it instead of their primary source, preventing you from being buried under a load of results from the primary source.

Now, what you read above was what I consider to be the important stuff. If you're feeling that you've had quite enough information about the popularity of different engines already, don't feel guilty about not reading the stuff below. Why? Because next we're going to take a look at non-independent engines, that means engines that don't have their own database. These engines have no individual content and cannot be submitted to, so you shouldn't worry too much about them. But if you have the desire to know what databases are used most, well, you're going to have to continue reading..

Looks like you've made up your mind, so here's a chart of visitors to non-independent engines:

As you can see from the chart, non-independent engines get massive traffic too. But remember that searching is usually not the primary feature of these sites. An independent engine probably gets searched more than a non-independent one even if the number of visitors is the same.

As with the previous chart, take these figures with a grain of salt. These figures can include visits to parts of the site that do not have a search feature, ie. AOL's figures may include the homepages of their users. Anyway, lets proceed and look at who uses who's database:

• AO: AOL. A big ISP and a big site according to the numbers. When it is searched, it begins by showing up to three 'Recommended sites', if they are available. These consist of AOL's own sites and paid advertisements. AOL then continues by giving out three 'Sponsored links', which come from GoTo. After those, it gives out matching sites from the ODP directory, but ranks them differently than the ODP itself. It seems that AOL spiders and indexes ODP sites, unlike ODP, which uses only the title, URL and description provided to it. This means that if your ODP description/title/URL doesn't contain the word "promotion", you won't be found on ODP when that word is searched. But if one or more of your site's pages contain that word, AOL will see you as relevant even when ODP doesn't! Relevant ODP categories are also shown at the bottom of the results page. If the ODP site results get exhausted (they rarely will, unless the keyword used is rare), it shows relevant sites found in the Inktomi database.

• MS: MSN. Bill's portal has become one of the biggest players in the search engine industry. Their search function will first give out a few "featured sites", if any are available for the keyword searched. These are mostly paid advertisements and Microsoft's own sites. After that, it will browse through Looksmart's directory, showing any relevant sites in the directory. When every match found there is shown, it will use the Inktomi database.

  Note that if the keyword searched is of adult nature, there is a small exception to what is described above. Before showing any results, the user is asked if he/she wishes to use a certain adult search engine or to see normal MSN results like explained above. This naturally means that people who run adult sites will see less hits from MSN, as some users will decide to use the other engine instead.

• NS: Netscape. When searched in the default mode, Netscape uses "Netscape Search". It first gives out two results from GoTo under "Partner Search results", then gives up to three relevant results from within the Netscape site and goes on to give 10 related ODP categories. After those it shows relevant sites from the ODP directory. If no matching ODP sites are found, it uses Google, but it does not show Google results if there is even one matching ODP site. Netscape can also be told to search from Excite, Ask Jeeves, Google, GoTo, HotBot, Looksmart, Altavista or Lycos databases. However, I believe that an overwhelming majority at least begins their search by using the "Netscape Search".

• IW: Iwon. First gives out a maximum of three relevant categories found in Looksmart, then uses Inktomi. The user also has the option to select a related 'popular search' powered by Direct Hit from the right side of the screen.

• RL: Real. Their small search box is located at the bottom of the front page, thus I believe that it is only used by a small percentage of visitors. Users may search either the Real site or the NBCi database (default). When the NBCi database is searched through Real, it uses a special "RealPlayer mode", which gives sites that support RealPlayer according to the NBCi Directory & LiveDirectory at the top. Sites that do not support RealPlayer are ranked very low.

• ZD: ZDnet. Users may search either ZDnet (default) or "The Web". The latter first gives out three top matches from GoTo and then uses CNet's Search.com metasearch engine, which takes its results from Altavista, Direct Hit, Findwhat, GoTo, NBCi, Open Directory, Yahoo, Lycos and many other engines.

• IS: Infospace. IS carries the Looksmart directory on its front page and also has a search feature. In the default mode (Web Search) Infospace uses the Dogpile metasearch engine, which draws its results from several big search engines. The search feature can be changed to use GoTo, Looksmart or Open Directory, if the user so chooses.

• IC: ICQ. The search feature is buried at the bottom of the front page, and can be used in two ways. The default mode starts by delivering related categories from the ODP, then gives out matching sites from ODP. It seems to use a similar method in ranking ODP results as AOL, although ODP titles and descriptions seem to get a lot of weight in the algorithm. After all ODP matches are shown, the user may click on a link to see additional results from Inktomi. The other mode searches the ICQ site for matches.