:: Search Engine Promotion ::

:: Part 4 - Submitting to Crawlers (Google, Inktomi, FAST, Teoma & AltaVista) ::

Crawler-based search engines automatically visit web pages to compile their listings. This means that, unlike directories, you are likely to have several if not many pages listed with them. This also means that by taking care in how you build your pages, you might rank well in crawler-produced results.

Optimizing pages for crawlers is covered more in the Optimizing For Crawlers section of Search Engine Watch. If you have time, it is recommended that you read the pages in this section, especially the Search Engine Placement Tips page. However, by simply following the submission tips below, you can at least get your pages listed with crawlers, where they might naturally rank well for certain terms.

Submitting To Google

Arguably the most important crawler-based search engine is Google, because both the important services of Yahoo and Netscape use some of Google's results. In addition, many people also go directly to Google to search.

The absolutely best way to get listed with Google is to build links to your web site. Indeed, this is the best way to get listed for free with all the major crawlers listed on this page. Crawlers follow links, so if you have good links pointing at your web site, the crawlers are more likely to find and include your pages.

Here's the good news: if you submitted your site to the major directories and got listed with one or more of them, then Google and other crawlers will almost certainly pick up the URL that was listed. This means you may not need to do additional work to get listed with crawlers.

Nevertheless, you may find it helpful to do some link building beyond the directories.

Aside from link building, Google does provide an "Add URL" page that lets you submit a URL directly to its crawler:

Add Your URL To Google
http://www.google.com/addurl.html

There's no guarantee that Google will actually include a URL submitted to it this way, however. Despite this, it makes sense to submit your home page and perhaps one or two other URLs from "inside" your web site via the Add URL page.

You really don't need to submit more than this. The only reason for submitting some of your inside pages is in case there is a problem reaching your home page. This gives Google an alternate route into your site. From whatever page it visits, it will look for links to other pages that you have and perhaps include those. This is true for other crawlers, as well.

If you have a brand new web site, it will probably take about a month before Google lists your web pages. Because of this, you might consider making use of its paid placement program, which is covered in the next part of this guide.

Submitting To Inktomi

Inktomi is an important crawler-based search engine because it provides some of the results used by major search sites such as MSN Search, AOL Search and HotBot.

As covered with Google, building links is the best way to get listed for free. Also, as with Google, Inktomi does take submission via Add URL pages. It doesn't maintain its own, however. Instead, you need to submit using one of its partners. I recommend using the HotBot UK Add URL page:

HotBot UK Add URL
http://www.hotbot.lycos.co.uk/submit.html

Exactly as with Google, you need only submit your home page and one or two inside pages. Also, as with Google, you can expect a delay of one month or longer until your pages show up, assuming they are accepted. Acceptance is not guaranteed to those using the Add URL page.

In contrast, Inktomi will guarantee to include your pages if you use its paid inclusion program. In addition, it will get your new page listed within 2 days. Because of this, for a brand new site, it's well recommended to use the program. You can quickly get yourself represented in Inktomi for a modest amount.

You'll want to use the "Search Submit" program to get at least your home page listed, for a $39 fee. This also means the page will be revisited regularly, for up to a year. After this time period, if you don't renew, your page might possibly be dropped by Inktomi. However, if you've built some links, this is highly unlikely. Home pages are deemed as very important by all crawlers, and it would be unusual for these crawler-based search engines not to include them for free. Paying the fee is really only speeding up the process of getting the home page listed, for a brand-new site.

The Search Submit program isn't sold by Inktomi but rather through its partners, all of which are listed below:

Inktomi Search Submit Partners
http://www.inktomi.com/products/web_search/submit.html

You can also choose to submit many more of your pages through the program, and the fees are less than for your first page. In addition, there's an "Index Connect" program for those who have thousands of pages they wish to pay to include.

Submitting To FAST Search

FAST Search is an important crawler-based search engine primarily because it powers some of the results for the popular Lycos service. It also maintains its own search engine called AllTheWeb.

Again, as with the other crawlers covered so far, building links is the best way to get listed for free. And again, as with the other crawlers, FAST does take submissions via an Add URL page:

Fast/AllTheWeb Add URL Page
http://www.alltheweb.com/add_url.php

As with the other crawlers, you need only submit your home page and one or two inside pages. Expect a delay of up to six weeks until your pages show up, assuming they are indeed accepted. This is not guaranteed to those using the Add URL page.

Like Inktomi, FAST will guarantee to include your pages if you use its paid inclusion program. It will get your new pages listed within 2 days. As with Inktomi, it's recommended that new sites use the program to get their home pages quickly listed.

The cost is $30 and means that the page will be revisited every two days, for up to a year. As with Inktomi, it's possible that your page could be dropped, if you don't renew. However, again as with Inktomi, this is highly unlikely to happen to a home page.

FAST doesn't sell paid inclusion directly to the public. Instead, inclusion is sold by FAST's portal partners. Currently, the best way to sign up is via the Lycos program, below:

Lycos InSite
http://search.lycos.com/searchservices/

Submitting To Teoma

Teoma is an important crawler-based search engine because it powers some of the results that appear at the popular Ask Jeeves web site. In fact, Ask Jeeves owns Teoma.

Unlike the other crawlers covered so far, Teoma has no free Add URL page. This doesn't mean that you can't get listed, however. Teoma crawls the web, so if you have links pointing at your web site, you may get included naturally.

Basic tips on building good links are covered on the Search Engine Placement Tips page, while the More About Link Analysis page available to Search Engine Watch members provides in-depth advice on building relevant links to your web site. Consider reading what's covered on one or both of these pages, to help yourself with Teoma.

Like Inktomi, Teoma will guarantee to include your pages if you use its paid inclusion program. It will get your new pages listed in a week. As with Inktomi, it's recommended that new sites use the program to get their home pages quickly listed.

The cost is $30 and means that the page will be revisited each week, for up to a year (and currently, for 15 months through a special promotion). As with Inktomi, it's possible that your page could be dropped, if you don't renew. However, again as with Inktomi, this is highly unlikely to happen to a home page. To sign-up for the program, visit the page below:

Ask Jeeves/Teoma Site Submit
http://ask.ineedhits.com/

Submitting To AltaVista

AltaVista is an important crawler-based search engine because, while its popularity has dropped over the past year, significant numbers of people still use it to search the web.

As with the other crawlers covered so far, building links is the best way to get listed for free. Also, as with the other crawlers, AltaVista does take submission via an Add URL page:

AltaVista Add URL Page
http://addurl.altavista.com/sites/addurl/newurl

As with the other crawlers, you need only submit your home page and one or two inside pages. Expect a delay of one month or longer until your pages show up, assuming they are indeed accepted. This is not guaranteed to those using the Add URL page.

Like Inktomi, AltaVista will guarantee to include your pages if you use its paid inclusion program. It will get your new pages listed in about a week to two weeks. As with Inktomi, it's recommended that new sites use the program to get their home pages quickly listed.

The cost is $39 and means that the page will be revisited regularly, for up to six months. As with Inktomi, it's possible that your page could be dropped, if you don't renew. However, again as with Inktomi, this is highly unlikely to happen to a home page. To sign-up for the program, visit the page below:

AltaVista Express Inclusion
https://www.infospider.com/av/app/

As with Inktomi, you can also choose to submit many more of your pages through the program, and the fees are less than for your first page. In addition, there's an "Trusted Feed" program for those who have thousands of pages they wish to pay to include.

(* all prices discussed are in US funds)