There have been major and significant changes in the search engine industry over the past few years, with surprising announcements almost every month. While most of the search engines' domains remain active, the actual state of the search engines is quite different. The URLs that used to lead to a unique database with unique search features and capabilities may or may not still do so.

And like so many aspects of the Internet, the death of a search engine is no simple matter. It can come in a variety of styles. Indeed, most of the original search engine URLs remain, and with some kind of a search box on the page. Many of our organizations continue to maintain Web pages with links to search engines. Which ones should we still link to and which have really died? It helps to understand how search engines die and which are left standing.
 

SEARCH ENGINE MORTALITY

In some ways, all the old search engines are still with us. The nature of the Internet and its URLs leads to a strange life after death for many search engines. At a minimal level, any moderately popular search engine URL is unlikely to die. Someone will want the traffic that the old URL will bring in.

So what happens to the old URL? It can simply be redirected. This is the case with the old SavvySearch metasearch engine. Bought out by CNET, the old www.savvysearch.com simply redirects to CNET's www.search.com site. It is an instantaneous redirect, at least on a fast Internet connect.

Alternatively the old URL can remain as a separate destination, even branded with the old logo. Yet the underlying database and search technology may well be completely different. This is what happened with WebCrawler. Originally a completely separate and popular search engine, it was bought by Excite years ago. For several years, Excite continued to maintain WebCrawler as a separate search engine. WebCrawler had its own spider and its own search features. Then, last year, Excite suddenly changed it to the Excite database. It was still labeled as WebCrawler, with the WebCrawler logo at the WebCrawler URL, but the original WebCrawler search engine was dead.

A few search engines have completely died. Inference Find, a well-respected metasearch engine, used to be located at www.infind.com, a URL which now results in an "address not found" (404) error message. The company shut the site down because it could not find a way to make enough money to support it. Contentville also died and shut down its URL. Although both of these URLs are currently dead, some other company could pick up the domain names in the future and resurrect the address. But beware: Once the domain is bought by a new owner, any kind of content could be placed at those URLs, either related to the older search engine or not.
 

OVERTURE

The most common way for a search engine to extend its life, at least as a specific Web site, is to give up its own database and search system in place of someone else's. The most common candidate is one that pays. An example is Overture. Formerly known as GoTo, Overture is a pay-for-positioning search engine. Advertisers can bid a certain amount for specific keywords and have their site appear above sites that bid lower or not at all.

One way to think of Overture is as a search engine with two distinct databases. The first is the paid links; those pages for which the site owner has agreed to pay a certain amount for every click through. This database of paid ads is supplemented by a larger, more traditional search engine database from Inktomi.

When a search engine dies and retires its own database and spider, Overture is an easy replacement. In part, this is because Overture can work out a revenue-sharing agreement with the old search engine. So the dying search engine does not have to do any work to build a database and is offered a way to make money. The search box on its page can look the same as before, but it may give completely different results, with the paid results coming up first. It can even be set up so that only paid results show up and no Inktomi results are available. As search engines die off, Overture is often what is left in their place.
 

THE SEARCH ENGINE GRAVEYARD

So who is on the dead list? One of the earliest search engines fatalities was the Open Text Index, the first search engine to introduce field searching capabilities. In early 1998, the Open Text Index was replaced by a business-oriented search engine known as Livelink Pinstripe, which then changed several times before it finally expired for good. One of the many URLs for the original Open Text Index, www.opentext.com:8080, is long dead, yet a quick link search at AllTheWeb.com shows that thousands of Web pages still link to that dead URL with its 8080 port number.

Another old-timer, Magellan, is dead with no logo or URL remaining. The old Magellan URL of www.mckinley.com(a URL that never made any sense to me anyway) now automatically redirects to WebCrawler, another dead search engine. Both Magellan and WebCrawler were bought out by Excite in days gone by. At least the WebCrawler site still has its old look and feel with the old WebCrawler logo.

But its parent search engine, Excite, is now dead. It not only died, Excite@Home took the bankruptcy route. The Excite.com domain remains, bought out by metasearch engine owner Infospace for the traffic that it can still bring in. Users going to any of the dead Excite search engines (Magellan, WebCrawler, and Excite) should wonder what search engine they really are using. At this point, it is the search engine life extender—Overture. And in all three cases, the results are first the Overture advertising database followed by the Overture results from Inktomi, with no obvious differentiation between the two.
 

SLOW DEATH OF PORTALS

The next stop in the search engine graveyard are the portals. Back during the Internet's heyday, everyone wanted to be a portal—a site that offered search, news, e-mail, and every other common service you could think of. The portals attracted many users and corresponding advertising dollars. Some of the portals had their own search engine databases while others provided some unique search features.

Infoseek started as a search engine, but after being bought out by Disney, it became a part of the larger Go portal. Infoseek remained the underlying search engine until Disney gave up on it as a portal. Go still exists, and the infoseek.com domain redirects to it, but what does it actually search? Overture, of course, with both the ad database and Inktomi.

Another media-sponsored portal went a similar route. Snap, which became NBCi after NBC bought it, used to have its own Web directory, along with a unique treatment of the Inktomi search engine. The directory died outright. The search box now uses Dogpile, a metasearch engine acquired by InfoSpace. With up to 10 results from each, Overture, Looksmart, FindWhat, Sprinks, and About.com are the first five search engines shown. FindWhat and Sprinks are paid-positioning search engines like Overture.

iWon made a name for itself as the portal to use for getting entries into cash giveaway sweepstakes. In the past, iWon had used the largest available Inktomi database and also combined search results from databases such as LookSmart, Fact City, and others. And while iWon is still giving away money, it has backed away from its more advanced Inktomi and other search capabilities. Its primary search results are from the Overture ad database, but those are followed by results from the larger Inktomi database, not just the smaller Inktomi database from Overture that other portals use. But Fact City results are gone and the emphasis is certainly on the ad results first.

As the eldest and best known of the portals, Yahoo! never had its own search engine. Its core directory remains, although it changed its follow-up search engine from AltaVista to Inktomi to Google. And although its stock has dived, Yahoo! is nowhere near death. It is still a hugely popular site for search as well as for its many portal offerings. Yet even Yahoo! has added "Sponsor Matches" from Overture's ad database for additional revenue and to help Yahoo! remain alive.
 

NORTHERN LIGHT

Most disappointing to many librarians was the recusal of Northern Light from the Web search engine scene early this year. The combination of Northern Light's Web database and its Special Collection of published articles made it a unique search engine. Its folder-based organization of results and advanced search features made it especially attractive to information professionals.

With its death as a Web search engine, it continues to provide enterprise services and public access to its Special Collection, including its Current News search and Special Editions. And it has avoided the common practice of delivering Overture ads as results.

It was not surprising that a few days after this shutdown, divine, inc. acquired Northern Light. The press release said that divine was acquiring "certain assets of privately held Northern Light," and it was obvious that in today's Internet economy, the public Web search engine was seen as a liability rather than an asset.

Although Northern Light is no longer accepting submissions, its crawler is still out there, and the search engine will be maintained for enterprise clients. In addition, some of its Special Editions have links to searches that include Web content. Yet for most practical purposes, Northern Light is now dead as a Web search engine for the general public.
 

OTHERS IN THE GRAVEYARD

Nor are these the only graveyard occupants. The old Lycos database is long gone, although the site remains as an active and viable portal and search engine. However, its search engine database and search features now come from Fast Search. The old spider has been retired and the old database completely replaced.

Both AltaVista and Remarq let their Usenet news databases die, leaving only DejaNews as a Usenet news search engine. Then when DejaNews died, Google scooped up the leftovers and launched Google Groups, which eventually became an even larger archive than Deja's.

Current awareness services like javElink, later known as EOMonitor, are now dead and gone from the realm of free search tools. News search engine connected with dead search engine, like Excite's NewsTracker and the old Infoseek News, are gone as well.
 

THE SURVIVORS

With all these discussions of the dead search engines, it is equally important to remember the living. Not only do many of the old search engines remain as some kind of search-related Web site, but also there are still several search engines surviving. Google, AllTheWeb, AltaVista, Teoma, and WiseNut all still survive with unique databases and features. So does Inktomi, available at partner sites such as HotBot and MSN Search and iWon, and in a slightly different version at each. Lycos also survives, using the Fast Search database that is also available at AllTheWeb. So despite all the turmoil and the passing of some favorites, the remaining search engines continue to fill a major role in information retrieval on the Net.

Comments? Email the editor at marydee@infotoday.com.