Skip to main content Accessibility help
  • Print publication year: 2017
  • Online publication date: June 2018

5 - Directory, clustering and similarity search engines



In the 1990s this class of search engine was extremely popular, and Yahoo! was leading the charge. It's hard to believe that at one point Yahoo! was worth over US$100 billion ( and it was a huge brand, closely associated with search and its search directory. It acted as a portal to the internet when there really wasn't very much organisation at all. Websites were springing up at the rate of thousands per day and early engines such as AltaVista simply couldn't keep up with this flood. Rather than try and seek out all of these websites Yahoo! and others like them took a different approach – web authors came to them, looked through their search directory hierarchy and found an appropriate niche for their site. This would then be submitted and once approved, the website would be included in the directory. The portal approach used directory search to encourage people to visit and to stay on the website. Coupled with news, video and messaging services it was hoped that a compelling offer would ensure that consumers wouldn't look anywhere else.

However, this failed to take into account the fact that people wanted to search more quickly and effectively, and by tying advertisements to the type of searches that people were running Google was able to take an increasingly larger share of the marketplace. The growth of Facebook, Twitter and YouTube meant that people were also getting their news from various different places, and communication tools such as Messenger and WhatsApp further depreciated the value of the portal approach. Finally, because of the sheer amount of growth of websites, and importantly the increase in technology, speed of spidering websites and fall in storage costs meant that free-text engines were able to index far more, and more effectively, than directory-based engines. As users deserted the sinking ship of Yahoo! and similar sites, advertising revenue decreased and it simply wasn't possible to change course and go from a directorybased to a free-text approach, although Yahoo! did attempt that with their ‘Yahoo!-Microsoft Search Alliance’ in December 2009. In late December 2014 Yahoo! closed its Directory offering, 20 years after it began.