Introduction

This chapter examines developments in systematic reviewing and what they mean for the search process. It introduces recent methods of synthesis, including mixed methods reviews and realist syntheses. It goes on to consider how innovative review types affect how we plan and execute search strategies. The chapter explores the value of other question formats and why you should choose a method appropriate to your review question. It starts by examining acknowledged limitations of the conventional systematic review model, both within health and multiple disciplines in which systematic reviews are gaining traction.

Challenging the conventional systematic review model

Since the mid-1990s, the conventional systematic review model, which assumes a closely focused question, a comprehensive search and a focus on synthesising ‘stronger’ rather than ‘weaker’ evidence, has proved remarkably flexible and resilient (Petticrew, 2015). The model has survived transfer to other emerging domains for systematic reviews, such as education, management and environmental science. However, this model faces renewed challenges as decision-makers seek to address ever more complex questions and to explore complex socially-mediated interventions, requiring diverse data and approaches to synthesis, and consequent changes to how studies are identified and selected.

Petticrew characterises the current era as moving from ‘What works’ to ‘What happens’ (Petticrew, 2015). Specifically, new forms of knowledge synthesis are required to explore people's perceptions of their situation, to identify underlying theories to explain how an intervention, policy or programme works, or to understand what makes an intervention, policy or programme more or less likely to be implemented (Tricco et al., 2016a). This, in turn, requires ‘other types of search approaches (e.g. snowballing of articles, focusing on identification of key theories)’ (Tricco et al., 2016b, 20) as well as others covered in this chapter.

The time and cost associated with producing systematic reviews offer a further challenge to their use in supporting decision-making (Tricco et al., 2017). Technical aspects of the search process are challenged by the rapid review movement which has reinterpreted ‘How far should you go?’ (Ogilvie et al., 2005) as ‘How little searching is enough?’ (Booth, 2010). One inevitable result is that three assumptions of the ‘systematic review catechism’ – the question, the comprehensive search and the privileging of rigour – are all receiving critical scrutiny.

Introduction

Paradoxically, with altmetrics in the ascendancy, the scholarly world is witnessing a renaissance of interest in traditional metrics. This chapter examines long-standing and recent drivers for metrics from the complementary perspectives of scientists, research organizations and funding agencies. It outlines established metrics for individuals (e.g. citations and the h index) and for teams and journals (the journal impact factor and other proprietorial alternatives). It examines the extent to which these metrics correlate to other important characteristics such as newsworthiness and journal prestige. The h index is an author-level metric which scores academics by their most cited papers. For example, if an academic has ten papers that are each cited ten times, their h index is 10. If they have ten papers cited no more than once each, their h index is 1.

The chapter considers challenges raised by the emerging ‘impact agenda’, including a need to capture ‘impact on society, social impact, real-world impact, knowledge translation, and uptake by the public’ (Eysenbach, 2011). It also describes how both traditional and impact metrics have been used in a variety of research and performance management contexts, giving examples of appropriate and inappropriate use.

The chapter examines criticisms of established measures and how these criticisms might, at least in theory, be addressed. It discusses manipulation and game playing which exploit acknowledged weaknesses of metrics, and provides high-profile examples. It examines the extent to which relatively recent forms of publishing, such as open access journals, are accommodated by, or pose challenges to, traditional metrics.

The chapter sets the scene for the considered evaluation of altmetrics that follows in subsequent chapters. It concludes by examining whether traditional metrics still occupy a role within a world that is increasingly populated by social media and social networks.

On metrics and madness

In Prokofiev's Lieutenant Kijé, a slip of a clerk's pen on a list of officers for promotion compiled for a mad tsar leads to the creation of the fictitious hero. Fearful of displeasing the mad tsar, his advisers manufacture increasingly elaborate escapades for the imaginary Kijé, each one rewarded by a successive promotion.

Introduction

Recent years have seen an increasing imperative for health librarians and information professionals to demonstrate that they possess, and use, mechanisms for ‘keeping current’ (Health Executive Advisory Group, 2004; Cunningham, 2010). As the speed of organizational and technical change becomes increasingly rapid, the requisite skill base covers an ever wider range of competencies (captured in the acronym COMPLIANT). Increasing user expectations, coupled with demands for quality, accountability and efficacy of practice, further emphasize the need to demonstrate strategies for periodic refreshment of skills and knowledge (Lacey and Booth, 2003). At the same time, conferences and workshops, traditional routes for acquiring such updates, are becoming less accessible as budgetary constraints and lack of time, combined with the need to provide continuity of front-line service, limit opportunities for staff to attend external training courses. E-learning, delivered via the web and facilitated by an increasing array of Web 2.0 technologies, offers the prospect of delivering staff development to library staff in the workplace (Ayiku et al., 2005).

Requirements for staff development

Our review of workplace-based e-learning identified five major themes: (1) peer communication, (2) flexibility, (3) support, (4) knowledge validation and (5) course presentation and design (Booth et al., 2009). The implications of these themes are as follows:

How Web 2.0 measures up

It is immediately apparent that Web 2.0 technology has much to offer in meeting the five identified needs of workplace-based learners rehearsed above. The interactivity and ease of use of blog and wiki technologies facilitate communication with (Grassley and Bartoletti, 2009) and peer support from fellow learners (Moen et al., 2009).

Introduction

Information access and retrieval has generated significant research output over recent decades. This chapter examines issues in evaluating information retrieval systems and techniques. It then identifies issues involved in practical use of such systems. Sources from the information science and computing literature feature in addressing such issues. Illustrative studies are assessed for quantity and quality. The chapter concludes by describing priorities for future research. The chapter is followed by two special topics examining the research evidence on searching aptitudes and characteristics, and the merits of different interfaces.

What is evidence-based information access and retrieval?

In their fifth domain, Information Access and Retrieval, Crumley and Koufogiannakis (2002) consider ‘creating better systems and methods for information retrieval and access’. Information retrieval (IR) is: ‘the science and practice of identification and efficient use of recorded media. Although … traditionally concentrated on the retrieval of text from the … literature, the domain over which IR can be applied effectively has broadened considerably …’. (Hersh et al., nd).

Asking the question

Budd and Miller (1999) highlight three particular questions from this domain:

Evaluation of information technologies provides a particular challenge to evidencebased information practice. Frequently, innovative technologies are adopted uncritically as a ‘good thing’ and evaluation viewed as an unwelcome ‘brake’ to technological advance.

Marcum (2003) identifies several key questions for the digital library:

The developing information environment and changing expectations and demands of library users have blurred the distinction between collection management and information access and retrieval, where digital libraries are concerned. Indeed, the third question, from the education domain, suggests that a holistic approach to question identification and subsequent resolution may be preferable. Such an interpretation is supported by Marcum's observation that we need to learn a lot more than we now know about the use of digital resources, their preservation, and the training needed for operating the library of the future.

Introduction

This chapter outlines the main components of collection management within the increasingly popular concept of the ‘hybrid’ collection which brings together both paper and electronic sources. It identifies a range of questions that management of such collections needs to address. The evidence base is characterized in terms of useful sources and important research studies. This chapter is followed by a special topic examining the research evidence on managing the electronic/ paper mix in connection with journal holdings.

What is evidence-based collection management?

The third domain identified by Crumley and Koufogiannakis (2002) is Collections by which they refer to the activity of: ‘building (and we could add maintaining) a high-quality collection of print and electronic materials that is useful, cost-effective and meets the users’ needs’.

The broad term ‘collection management’ includes collection development, maintenance and withdrawal. Collection development involves ‘planning, goalsetting, decision-making, budgeting, and acquiring materials and evaluating them’ (Gessesse, 2000). Richards and Eakin (1997) emphasize the centrality of this domain to information practice: ‘The library's collection is its heart because it is the collection that is the central information resource upon which most library activities rely …. Today's vision of the collection must incorporate both a broader range of materials … and a greater sense of connectivity’.

The first impression on reviewing this domain is the tremendous spectrum encompassed, from management-oriented tasks of collection development to the physical (or even electronic) sciences involved in preservation or restoration of printed or electronic materials. Some observers identify the specific field of archive management as one of the most unequivocally ‘evidence-based’ areas of information practice. On the other hand, Scott Plutchak, in reviewing collection development, affirms that ‘as with so much else in librarianship (and medicine, for that matter), [it] remains more of an art than a science’ (Plutchak, 2003). If evidence-based information practice is the science of making decisions then our view of evidence-based collection management has to be reconciled with the observation of Richards and Eakin (1997) that: ‘Librarians make decisions about collections …. Some claim these decisions are primarily an art; others argue that they are, or should be, grounded in scientific methods’.