Introduction

The rich, multi-dimensional, multi-modal data channel offered by multimedia brings with it unique challenges for information retrieval. The old saying equates a picture to a thousand words. While this is true for the amount of information that a picture can convey to a user, from a data management and information retrieval perspective the equation is not so straightforward. More than a thousand words and more complex representations are often needed to identify relevant media and assist users in their data finding missions successfully.

Traditionally digital and analogue multimedia documents have been indexed and queried using text descriptions in the form of metadata. Metadata for multimedia can be modelled or classified in a number of ways (Boll, Klas and Sheth, 1998; Troncy et al., 2007). It is useful to group multimedia metadata into four rough categories, as shown in Figure 13.1 overleaf, arranged by the level of human effort potentially required to create the descriptors, to illustrate the concept of the ‘semantic gap’. The semantic gap in multimedia information retrieval describes the difficulty in automatically generating usable high-level semantic descriptions (e.g. dog, party, ocean and so on) from automatically extracted low-level media features such as colour, texture, tempo, pitch and so on. The concept originates in psychology (Finke, 1989) but its use within multimedia information retrieval is defined by Smeulders et al. (2000) as ‘the discrepancy between the information that one can extract from the visual data, and the interpretation that the same data has for a user’.

When managing high volumes of multimedia data, it is very attractive to try and generate quality semantic annotations from features automatically extracted from the pixels or bits. Not only is this cheaper and more efficient than manual human annotations but it also improves the consistency of the annotations by reducing the influence of user opinion. Bridging the multimedia semantic gap is the target of a large field of research. We will discuss some aspects of it in the section ‘Piggy-back text and automatic annotation’ in the context of automatically annotating media; good general starting points can be found in Smeulders et al. (2000), Zhao and Grosky (2001) and Hare et al. (2006).

Introduction

Multimedia here refers to images, audio and video. Multimedia information retrieval means the process of searching for and finding image, audio and video documents through a multimedia search engine. This chapter will take image retrieval as an example to discuss interaction models and interfaces developed for multimedia information retrieval and to illustrate information-seeking behaviour in relation to image search.

Current image search engines are mainly based on keyword annotations or information extracted from the image's context (e.g. web page text). This approach has three limitations. First, the manual annotation of images requires significant effort and thus may not be practical for large image collections. Second, as the complexity of the image increases, capturing image content by text alone becomes increasingly more difficult. Finally, it relies on the user being able to articulate and enter a text description of their information need using the same vocabulary (and language) as the text annotations.

In seeking to overcome these limitations, content-based image retrieval was proposed in the early 1990s (Rui et al., 1998). It searches using images rather than keywords as the query (discussed in more detail in Chapter 13, ‘Multimedia: information representation and access’). Content-based image retrieval systems have since been primarily used for image searches on collections with limited annotations, or for image searches where annotation is not required, such as trademark search (Eakins, Riley and Edwards, 2003). More recently, Google has launched a new application called ‘Google Goggles’ for Google Android mobile phones, which is a content-based search application and allows people to search for more information about a famous landmark or work of art simply by submitting a photo of that object (Jamaal, 2010).

A basic content-based image retrieval system interprets the content (e.g. colour, texture, shape and so on) of the images in a query and in the collection, calculates the similarity between the images in the query and the object images in the collection and ranks the object images in the collection according to their degree of relevance to the images in the query (Marques and Furht, 2002).