There are several very good sites with excellent content and information about search engines. This article is intended as a summary and background to search engines and I recommend further reading (see links below) to anybody who wants to be successful using search engines.
The first most important thing is to understand the different types of search engines. The term ‘search engines’ is used today to loosely refer to any site that offers a way to find other sites. But these sites all work very differently with varying degrees of success. You must know how the site works so that you can prepare your submission correctly.
The search engine landscape is changing rapidly with mergers, and new sites appearing. The information here is accurate at the time of writing, but will most likely change.
I will group the sites as follows, but there is no rule and you may see the same information promoted differently on other good sites.
Search Engines
True Search Engines are sites that are run completely by computers (bots/spiders). You simply submit your site’s address (URL) and then their computer will come to your site and index it into their database. You have very little control over when your site is indexed and the results you get depend very much on the text on your site and how the search engine works. It can take up to 8 weeks for a bot/spider to come to your site. Make sure your HTML, Meta-Tags and content are well formatted before submitting.
Because your site is read and indexed by a computer it makes sense that the way your site looks has no effect on the success of your listing. In fact sites with mostly text and no graphics will often do much better. Flash objects, Java Applets, JavaScript and Frames can often work against you and cause the bots to rank your site low, or even reject your site completely. I strongly recommend you go to http://www.spider-food.net for some great information about how search engines work.
Each search engine ranks your site based on different criteria. Some place high importance on your domain name, some use the HTML tag, and some ignore your Meta-Tags completely. You need to read the guidelines are each site carefully to make sure you are doing the right things.
These are some of the best-known true search engines (with a direct link to their submission page):
- www.google.com – ( http://www.google.com/addurl.html )
- www.lycos.com – ( http://home.lycos.com/addasite.html )
- www.altavista.com – ( http://add-url.altavista.com/cgi-bin/newurl )
- www.alltheweb.com – ( http://www.alltheweb.com/add_url.php3 )
- www.hotbot.com – ( http://hotbot.lycos.com/addurl.asp )
- www.directhit.com – ( http://www.directhit.com/util/addurl.html )
- www.northernlight.com – ( http://www.northernlight.com/docs/regurl_help.html )
Meta Search Engines are search engines that get their content from other search engines. You can’t actually submit to them but it is worth knowing about them. Your success from these sites will depend directly on your success in other search engines.
Some Meta Search Engines:
- www.metacrawler.com
- www.megaspider.com
- www.dogpile.com
- www.mamma.com
Directories
Directories are sites that are managed mostly by humans. They are usually arranged in a hierarchical fashion. You must first choose a category/location in the directory and then write a title and description to be placed in the directory. It is very important that you choose the category, title and description carefully as this will determine how well your listing will work.
Because a human reviews your submission, it is important that your site is impressive enough to convince the reviewer to list your site. While the actual text and Meta-Tags on your site have no effect on search results, you need to make sure that there are no broken links, no ‘under construction’ signs, and that your site is clear, well organised and clearly contains useful content worthy of listing in a directory.
Yahoo and Looksmart now charge US$199 for an application from a commercial site (and they still charge you even if they reject your submission). Directories are difficult to get into, but they often bring the best results if done well. Take time to read the submission guidelines carefully.
These are some of the best-known directories:
- www.yahoo.com – Fees for commercial sites
- www.dmoz.org – Free
- www.looksmart.com – All Fee based
- www.goto.com – Pay per click model – https://www.goto.com/s/dtc/signup/
Some of the best-known search engine sites are actually powered entirely by other directories. You can’t submit your site directly to these sites although they will direct you to the appropriate site that powers them.
- www.msn.com – powered by www.looksmart.com
- www.netscape.com – powered by www.dmoz.org
- www.excite.com – powered by www.looksmart.com
- search.aol.com – powered by www.dmoz.org
- www.go.com – powered by www.goto.com
Even more confusing
As if to make this whole process even more confusing, many of the search engines and directories share their content through periodic contracts.
Right now, if you do a search on Yahoo and no result is found then Yahoo will automatically call upon Google to get more results, so a listing in Google may also help you in Yahoo. This was not always the case, Yahoo used to use www.inktomi.com and other search engines.
Google, in turn, uses www.dmoz.org to augment its listings so a listing in www.dmoz.org will help you in Google. Netscape, Lycos, and HotBot also use www.dmoz.org.
Several search engines also make use of www.directhit.com content. Directhit.com also uses www.dmoz.org content, and Askjeeves.com has recently purchased www.directhit.com.
There are also a few commercial sites that buy or develop their content. You generally can’t submit to these sites and have no control over any traffic they may send to you. These include www.askjeeves.com, www.about.com and www.infospace.com.
Keywords
Your choice of keywords is critical. There is no point in spending time and effort promoting your site to search engines and directories if you choose the wrong keywords to target.
The content of your sites, the titles and descriptions you choose in directories and even your domain name must be targeted to specific keywords.
This is a very complicated area and I will simply recommend that you go to http://www.wordtracker.com for more advice and a good solution to this problem. This site will help you determine what words are being searched for on each of the major search engines and it will also help you determine which words are the most effective to target for your area of business.
Volume of traffic
So which are the best search engines and directories? This is difficult to answer, but the following stats are relatively helpful.
Media Matrix Monthly Ratings of Search Engines – January 2001
KEY: YH=Yahoo, MSN=MSN, AOL=AOL, LY=Lycos, GO=Go (Infoseek),
NS=Netscape, NBC=NBCi, EX=Excite, LS=LookSmart, AV=AltaVista,
AJ=Ask Jeeves, GG=Google, iWN=iWon, GT=GoTo, DH=Direct Hit, DP=Dogpile.
Summary and Links
Of course once you digest all the above you will most likely have many more questions. What is a Meta-Tags? Where do I get details of them? How can I use frames and not cause search engine problems? Etc.
The links below will lead you to the answers to most questions. It is important to understand that the science of search engines is a relatively new one, and it is changing rapidly. You must work constantly to keep up and succeed. Most people will say, “search engines don’t work”, but this is because they haven’t taken the time to understand them completely.
A well-planned attack on the search engines will take you about 12 months to develop and implement completely. Some tasks will have good results immediately, but you must build up the process over a long period of time. Companies that started 12 months ago will be reaping the benefits now and will only have to do minimal work and study to hold their positions. You simply need to decide when you plan to start.
- http://www.spider-food.net – good technical background, easy to read
- http://www.webposition.com – excellent newsletter
- http://www.wordtracker.com – use this to select the best keywords
- http://www.searchenginewatch.com – good articles and statistics