Submitting To Search Crawlers: Google, Yahoo, Ask & Microsoft's Live Search



ClickZ
You are in the: ClickZ Network arrow

Searchenginewatch.com The source for search engine marketing
Members Area With Exclusive Content
Already a member? Enter Here Learn about the benefits:Enter Here
Departments & Info
Home
Latest Stories From SEW
Blog
News & Views
Discussion Forums
Ask & Answer!
SEW Experts
Expert Advice & Opinions
Search 101
Rating & Stats
Members Area
Exclusive Content
Get Feeds
About SEW
Advertise on SEW
Subscribe

Search SEW »

Members Area
Free Area
Search by date


Sponsored by

Search Engine News
From The SEW Blog
WebSideStory Rebrands as Visual Sciences
Google Launches New "Geo" Blog
Search Headlines & Links: May 9, 2007
Coverage from SEW Live -- Columbus
Social Media Marketing Do's and Don'ts
Search Marketing Expo is Coming Fast!
Yahoo and Microsoft: A deal made in heaven or hell?
Yahoo Enhances Travel Maps
Microsoft Takes Stake in CareerBuilder
Yahoo Auctions to Bid Adieu
Google Print Ads Now in AdWords Console
Travel Search Overview: Informative or Commercial?
Gates Plans to Spend Remaining Time Focusing on Search
User-Generated Videos On Your Domain?
Search Headlines & Links: May 8, 2007
More From Our Blog...

Search Engine News
From The SEW Forums
google search info and cookie
Evidence of Google Analytics increasing CPC
Directory sumbission
Could AdWords impact the U.S. Economy?
Ask Sponsored Listings to Launch Content Network
New domain or sub domain for new language site
Is adding selected pages to the sitemap enough to allow a full crawl?
SERPS - how to get sub-pages listed
New Keyword Selector Tool
Redirect to a Mobile Device
More From Our Forums...  
 

Search Engine Strategies Logo
Search Engine Watch's conference on search engine marketing comes to:
•   SES China: May 25–26, 2007
•   SES Milan: May 29–30, 2007
•   SES Toronto: June 12–13, 2007
•   SES Latino: June 18–19, 2007
•   SES Travel: July 26–27, 2007
•   SES San Jose: Aug 20–23, 2007
•   SES Local Search: Nov 2007
•   SES Paris: Nov 27–28, 2007
•   SES Chicago: Dec 3–6, 2007
•   SES London: Feb 19–21, 2008

Webmasters

print this article | e-mail a colleague | discuss this article

Submitting To Search Crawlers: Google, Yahoo, Ask & Microsoft's Live Search
› › › Webmasters

By Danny Sullivan | March 12, 2007

Part 4 of Essentials Of Search Engine Submission

Crawler-based search engines automatically visit Web pages to compile their listings. This means that, unlike directories, you are likely to have several, if not many, pages listed with them. This also means that by taking care in how you build your pages, you might rank well in crawler-produced results.

Optimizing pages for crawlers is covered in more detail in the Optimizing For Crawlers section of Search Engine Watch. If you have time, it is recommended that you read the pages in this section, especially the Search Engine Placement Tips page. However, by simply following the submission tips below, you can at least get your pages listed with crawlers, where they might naturally rank well for certain terms.

Submitting To Google

One of the most important crawler-based search engines is Google, because many people search there, plus it "powers" the main results of several other services (see the Search Engine Results Chart for an explanation of what "main" results are).

The absolute best way to get listed with Google is to get links to your Web site from other sites. Indeed, this is the best way to get listed for free with all the major crawlers listed on this page. Crawlers follow links, so if you have good links pointing at your Web site, the crawlers are more likely to find and include your pages in their databases.

Here's the good news: if you submitted your site to the major directories and got listed with one of them, then Google and other crawlers will almost certainly pick up the URL that was listed. This means you may not need to do additional work to get listed with crawlers.

Nevertheless, you may find it helpful to do some link building beyond the directories. Basic tips on building good links are covered on the Search Engine Placement Tips page. The Link Analysis & Link Building page available to Search Engine Watch members provides in-depth advice on building relevant links to your Web site. Consider reading what's covered on one or both of these pages.

The Submitting & Encouraging Crawlers page available to Search Engine Watch members also has advice on how your site architecture can be improved to naturally allow more of your pages to be added by crawlers.

Aside from link building, Google provides an Add URL page that lets you submit a URL directly to its crawler. However, there's no guarantee that Google will actually include a URL submitted to it this way. Despite this, it makes sense to submit your home page and perhaps one or two other URLs from "inside" your Web site via the Add URL page.

In addition to the ADD URL page, Google offers other options to submit a large list of URLs (a sitemap file) and verify ownership of your site within Google Webmaster Central for faster indexing, free of charge.

Once you have completed one of the options above, you really don't need to submit more than this. The only reason for submitting some of your inside pages is to give Google an alternate route into your site in case there is a problem reaching your home page. From whatever page it visits, Google will look for links to other pages within your site, perhaps including those in its index. This is true for other crawlers, as well.

If you have a brand new Web site, it will probably take about a month before Google lists your web pages naturally. Because of this, you might consider making use of its paid placement program Google AdWords, which is covered in the next part of this guide.

Finally, Search Engine Watch members have access to a detailed Guide to Google page that guides you even more through the process of submitting to the crawler and ranking well within its results. To learn more about becoming a member to access this information, visit the membership information page.

Submitting To Yahoo

Yahoo is an important crawler-based search engine because many people use the Yahoo search function, which provides the main results of several other services (see the Search Engine Results Chart for more about this).

As with Google, building links is the best way to get listed for free in Yahoo. Yahoo also offers a free URL submission form that you'll find listed on this page. Submit according to the same instructions as given for Google, above.

What if you aren't picked up for free? Yahoo has paid inclusion programs that guarantee to add the pages you submit quickly. The downside to these programs is that you'll be charged every time someone clicks on your listing. If you run out of money, potentially, your listing may be dropped. However, there's still a chance that even if you run out of money, you might continue to be listed for free.

Confused? You're to be forgiven, if so. The programs are so complex that we do not recommend beginners bother with them. Instead, there's a good chance that many pages in your site will just naturally get listed for free.

By the way, Yahoo's crawler incorporates technology from three different crawlers that it purchased in 2002 and 2003: Inktomi, AltaVista and FAST's AllTheWeb. However, any references you hear about those crawlers are superceded by the single Yahoo crawler.

Submitting To Ask

Ask.com search results have been powered by the Teoma ExpertRank algorithm since 2001 (see the Search Engine Results Chart).

Ask has no free Add URL page and no longer offers a paid site submission service. However, the Ask Jeeves/Teoma agent automatically crawls the Web, so if you have links pointing at your site from other sites, it will get included naturally.

Basic tips on building good links are covered on the Search Engine Placement Tips page, while the Link Analysis & Link Building page available to Search Engine Watch members provides in-depth advice on building relevant links to your Web site. Consider reading what's covered on one or both of these pages, to help yourself with Ask.

Submitting To Microsoft Live Search

Microsoft Live Search (formerly MSN Search) is an important crawler-based search engine used by many people. Formerly powered by Yahoo's crawler-based results, Live Search currently uses its own search technology, MSNBot, to automatically crawl the web looking for links within Web sites for indexing and ranking.

A review of Microsoft's search technology can be found in this article from Search Engine Watch: MSN Search Gets New Look; Microsoft Gets New Search Engine.

Next: Submitting Via Paid Placement Listings
Previous - Beginning

Editors' Note: The date shown on this article reflects updates provided by Claudia Bruemmer, Internet Marketing Writer and former ClickZ Managing Editor. This article was originally published by Danny Sullivan in July of 2004.

print this article | e-mail a colleague | discuss this article



Email this Article
View Printable Version






Search Engine Watch

www.searchenginewatch.com
Rebecca Lieb, Editor-in-Chief; Elisabeth Osmeloski, Editor

Having trouble? Tell us about technical questions or bug reports.
Let us know about newsletter problems or issues.

Incisive Interactive Marketing LLC. 2007 All rights reserved.
Legal Notices, Licensing, Reprints & Permissions, Privacy Policy