Health technologies such as drugs, diagnostic procedures or devices are routinely assessed in England and Wales for use in the National Health Service (1). Evidence submitted by the manufacturer to the Single Technology Appraisal (STA) process is independently assessed by external academic centers, known as Evidence Review Groups (ERGs) who review and critically appraise the evidence provided in the manufacturer's submission (MS) for its validity, quality, and generalizability (2–4).
Appraisal of search strategies is an integral part of the STA process and is carried out by the information specialist within the ERG team. Their role includes performing background searches for the reviewing team; critically appraising the search methods and strategies presented in the MS; verifying and reproducing the manufacturers’ searches; identifying any missing studies through independent iterative searches; and contributing to the writing of the final report by commenting on the strengths and weaknesses of the search methodology (1;Reference Guba5).
The National Institute for Health and Care Excellence (NICE) provides publicly accessible guidance for manufacturers on the finding and reporting of evidence (3;6). In brief, the manufacturer is required to describe their strategies for retrieving the evidence and justify their methods used in relation to the decision problem. Adequate details provided about the methods used and detailed strategies supplied in the appendices in the MS are necessary, so that searches can be reproduced by the ERG team.
An analysis of the strengths and weaknesses of submissions to the NICE STA has been conducted previously by two academic centers, but this did not examine in detail any comments on the quality of the manufacturers’ searches (Reference Carroll, Kaltenthaler, Fitzgerald, Boland and Dickson7–Reference Kaltenthaler, Boland, Carroll, Dickson, Fitzgerald and Papaioannou10). At present, there is no standard guideline for the information specialist to carry out the assessment of the MS searches and no study on the appraisal of strategies by ERGs in the STAs.
Despite this absence, there are other publicly available checklists on the appraisal of search strategies (Reference Sampson, McGowan, Lefebvre, Moher and Grimshaw11;Reference McGowan, Sampson and Lefebvre12), as well as recommendations on the reporting of searches in systematic reviews (13–Reference Higgins and Green15) and health technology assessments (Reference Busse, Orvain and Velasco16;17). However, reported guidelines for best practice are heterogeneous and not consistently applied by researchers (Reference Yoshii, Plaut, McGraw, Anderson and Wellik18) and there has been a call for a common approach to reporting in health technology assessments (HTA) (Reference Niederstadt and Droste19).
In this study, the ERG reports (published since inception until May 2011) were studied to define the limitations reported in the ERGs of the manufacturers’ search methods and strategies (what manufacturers do) with the purpose of developing a checklist for HTA practice in the United Kingdom and other countries. The key themes occurring in the ERG critique of both clinical and cost-effectiveness searches (what ERGs do) were identified and the extent of published checklists and recommendations used by ERG groups (how ERGs appraise searches) were considered here.
METHODS
ERG reports were obtained from the NIHR HTA Web site (accessed in May 2011) by Ruth Wong. Descriptive information about the STAs was obtained, such as: the year of publication; the academic group commissioned by NICE to undertake the appraisal; the availability of full text of the STA and publication in the HTA supplement; and the occurrence of ERG supplementary searching by the inclusion of strategies in the appendices.
The focus of this study is to identify the types of limitations as reported by the ERG groups on aspects of the manufacturer's search methodology and strategies. The search appraisal commentaries were often found within the “Clinical Effectiveness” or “Economic Evaluation” sections of the report preceded by the subheadings “Critique of the methods used by the manufacturer to systematically review clinical effectiveness evidence” (20). Supplementary search strategies by the ERG were supplied in Appendices if further searches were carried out for additional studies.
One reviewer (R.W.) extracted all the data from the search appraisal commentaries contained in the “Critique of the methods used by the manufacturer to systematically review clinical effectiveness evidence” sections found in the “Clinical Effectiveness” and “Economic Evaluation” sections of the ERG report. The limitations as specified in the ERG report were identified as clear statements of weakness in the search approach or strategy, errors identified or suggestions for further improvement. The focus of this study was only to consider what could be improved, rather than what was already being done well, so positive comments concerning search methods were not analyzed.
The authors chose an interpretive, inductive approach with principles of thematic analysis (Reference Miles and Huberman21). This approach is reductive, uses a “constant comparison” method and works by coding or classifying the extracted data into themes that capture or reflect those data, in this case, the limitations identified by ERGs in the searches in manufacturer submissions. The themes are, therefore, the first review author's own interpretation of the data with input from discussions with the second information specialist and reviewer.
A single theme might be the result of either small or large amounts of supporting data, with a common thematic basis, for example, “Language bias” or “Error in the use of PICO”. Similar themes were then “clustered” in a further process of data reduction to create twelve “main themes”. Some of these main themes, such as “Reporting” might capture as many as six of the original themes (see Table 2), while others, such as PICO, might capture as few as two or three. One reviewer (R.W.) carried out the primary analysis of the data.
In standard thematic analysis, the frequency with which a theme appears is less important than its actual appearance. In this study, however, the frequency of a theme's occurrence in the ERG report was also considered to be important. This was in part because the intention was to compare the frequency of particular identified, limitations across the clinical and cost-effectiveness elements of submissions. It was also because one aim of the analysis was the creation of a potentially usable checklist, which would seek to focus principally on those limitations that were both most frequent as well as most likely to introduce bias. After the limitations had been collated in the extraction table; the next step involved quantifying the occurrence of the limitation in a frequency table.
Based on the themes identified, a checklist was drawn up following discussion between the authors for recommendation in the appraisal of search strategies in the HTA manufacturer submissions.
RESULTS
STA Reports
There were eighty-eight ERG reports completed from 2006 until May 2011 (20). Only eighty-three (94 percent) of these reports were downloaded from the Web site. The remaining five (6 percent) were unavailable for reasons that are unknown. The number of published STAs varied considerably between 2006 and 2010, with 2009 having the most STAs (twenty-three reports) compared with the fewest STAs (nine reports) in 2008 (Figure 1). 49 (56 percent) of the 88 STA reports were published in a supplement to the HTA Monograph Series (22).
Figure 1. The number of published Single Technology Appraisals (STAs).
The ERG groups Liverpool Reviews and Implementation Group, School of Health and Related Research Technology Assessment Group and Southampton Health Technology Assessment Center have authored a total of forty-four (50 percent) of the ERG reports from 2006 until May 2011 (Supplementary Figure 1, which can be viewed online at www.journals.cambridge.org/thc2013122). Fifty-four reports (65 percent) show that some or all the manufacturer's searches were repeated by the ERG, whereas in twenty-nine reports (35 percent), the ERGs either failed to repeat the searches or it was unclear whether searches could be reproduced. Only twenty-five (30 percent) ERG reports provided additional strategies in the Appendix.
Themes
A total of twelve major themes were identified, of which, six relate to the search strategy (Tables 1 and 2). These are: strategy (overall strategy, PICO and terms, subject headings, free-text terms, syntax and structure); source; filters; translation; reporting and missing studies. It is worth noting that some of the themes are not mutually exclusive. For example, filters and translation relate to the strategy because they both involve the addition or adaption of terms between databases but they also represent crucial elements of the strategy distinct from the actual syntax and structure. Table 2 shows the frequency of occurrence of elements in the ERG report. Some ERG reports show variation in the number of limitations in the MS searches while others report few or none.
Table 1. Description of the themes
Table 2. Limitations identified from 83 ERG reports grouped into themes
Search Strategy
The search strategy theme contained the most number of types of limitations. The three highest occurring limitations were lack of use of or missing subject headings or thesauri terms and the omission of free-text terms including the use of synonyms.
Source
The omission of searching for unpublished data, ongoing or completed trials and conference proceedings was a frequent limitation by the manufacturer and reported in the ERGs. In addition, the omission of searches in other databases was also reported and recommended by the ERGs.
Limits
Even though the numbers reported are lower compared with the other themes, the inappropriate use of limits or unexplained use of limits to restrict the searches was the most frequently reported limitation.
Filters
Similar to the Limits theme, redundant application of a methodological filter to retrieve records of a specific study design was the most reported in this group of limitations. These include application of filters in the Cochrane Central Register of Controlled Trials (CCRCT), NHS Economic Evaluations Database (NHS EED) and EconLit databases.
Translation
There was inconsistent translation of free-text terms and subject headings and thesauri terms of comparators across databases and previously reported (published) strategies. These also include inconsistent field searching, use of filters and limits.
Reporting
In both reviews of clinical and cost-effectiveness evidence, unclear reporting of search details (such as dates, host platform, issue and date range) and missing strategies were the two most frequent limitations identified by the ERGs.
Missing studies
Twenty-one of eighty-three ERG reports identified additional studies either by repeating searches or by conducting independent searches in sources of ongoing or completed but unpublished research.
Clinical Versus Cost-effectiveness Critiques
Critiques for the two types of reviews are distinct sections within the ERG reports. The searches reported in the clinical effectiveness sections relate to searches for direct evidence (head to head studies), searches for indirect evidence and searches for adverse events (3;6). There were substantially more limitations in the clinical effectiveness review searches identified compared with cost-effectiveness (Table 2).
The cost-effectiveness searches include the retrieval of cost-effectiveness evidence, health-related quality of life studies, and research allocation and usage (3;6). Limitations that were confined to the cost-effectiveness review were, for example, reporting of quality of life (QoL) resources, cost searches, searches in specialist centers, economic terms, and missing QoL or utility terms. Thirteen ERG reports (16 percent) did not contain a critique section on the cost-effectiveness searches. The reasons for this are unclear. There were also fewer occurrence of missing studies identified by the ERG for the cost-effectiveness review compared with the clinical effectiveness searches. Only seven ERG reports (8 percent) stated the lack of systematization of searches for data used for the model.
The combined frequencies of the limitations reported by ERGs in both manufacturers’ clinical and cost-effectiveness reviews places “search strategy” theme above all the other themes in this study. In the clinical effectiveness review, the most frequently reported weakness is the limited use of synonyms and missing free-text terms in the provided strategies, as well a missing subject headings or thesauri terms. By contrast, the distribution frequencies of limitations between the themes are broad in the cost-effectiveness review.
The number of additional studies identified by the ERG was much higher (21 reports) in the clinical effectiveness compared with the cost-effectiveness review (5 reports). Further iterative searching had been carried out by the ERG teams to identify new studies in sources that the manufacturer failed to search such as ongoing and completed but unpublished research.
The distribution of limitations between themes is similar in both clinical and cost-effectiveness search critiques in the MS by the ERG. The ordering of frequency of occurrence of various limitations in themes for clinical effectiveness is as follows: Search strategy (n = 84), Reporting (n = 66), Source (n = 55), Limits (n = 22), Filters (n = 22), Missing studies (n = 21), and Translation (n = 15). By contrast to the cost-effectiveness: Search strategy (n = 53), Reporting (n = 45), Source (n = 18), Filters (n = 11), Limits (n = 8), Missing studies (n = 5), and Translation (n = 5).
Use of Checklists
Only four ERG reports (5 percent) acknowledged using the Peer Review of Electronic Search Strategies (PRESS) evidence-based checklist to critically appraise the manufacturer's search strategies. The PRESS checklist is used to evaluate electronic search strategies in systematic reviews and health technology assessment reports (Reference Sampson, McGowan, Lefebvre, Moher and Grimshaw11;Reference McGowan, Sampson and Lefebvre12).
Both NICE and other guidelines for systematic reviews stipulate that search strategies, search methods, and sources should be fully provided (3;14;Reference Higgins and Green15). However, manufacturers’ failure to report their strategies fully was the most common limitation found by the ERGs. None of the ERG reports reference any use of a checklist or recommendation guidelines for the appraisal of the MS reporting and presentation of the information retrieval process by ERGs.
DISCUSSION
The analysis of the ERG reports of manufacturers’ searches provides an insight into what manufacturers do in terms of searching, what ERGs do in relation to appraising searches and how they report on the appraisal process.
Manufacturers’ Searches
There are twelve themes identified from the limitations of manufacturers’ searches described in the ERG reports. The identification of few or no limitations in some ERG search critiques does not necessarily reflect systematic or comprehensive searching in the MS. Limitations in the manufacturers’ searches are primarily classified in the search strategy, reporting and source themes. Most notably, these relate to either omissions, such as failure to search in other databases, to search for unpublished studies, missing search details (sources searched, host platform and coverage dates) and missing full strategies. The former omissions relate to the comprehensiveness of the manufacturers’ searches, whereas the latter relate to the lack of transparency and careful documentation of the manufacturers’ search process which are common in both clinical and cost-effectiveness reviews. Transparency in the search process reported by the manufacturer is a key requirement for ERGs to repeat the MS searches. Although the availability of the source used to both the manufacturer and the ERG is a limiting step to reproducing and verifying the searches.
The search strategy feature is the main theme comprising six themes. The most frequent limitations are related to the failure to use subject headings or thesauri, missing free-text terms, errors in database syntax and errors in the combination of lines in statements. It is acknowledged that new medical subject headings and thesauri terms may be added to Medline, Cochrane Library, and Embase. The failure to use subject headings is at least equally and perhaps even more important than missing free text. Both of these will have an impact on the recall and precision of the search (Reference Sampson, McGowan, Lefebvre, Moher and Grimshaw11). Depending on the host platform, each database has a specific set of commands such as truncation and wildcard symbols and field tags. The main impact of truncation and wild-card errors is precision, whereas incorrect use of field tags will impact both precision and recall (Reference Sampson, McGowan, Lefebvre, Moher and Grimshaw11).
ERG Appraisal of Searches
The wide range of limitations identified in this study reflects the importance of the information specialists’ role within the ERG team in assessing the quality of the manufacturers’ searches. Information specialists design strategies for the comprehensive retrieval of the relevant literature. They are familiar with the specific commands in databases and responsible for documenting and reporting the search process, thus capable of identifying potential errors that might be found in the search process. Some of the limitations are errors whereby the main potential impact is on precision and/or recall, for example, limitations in the search strategy. Whereas other limitations are a matter of subjective judgment, such as whether the search was considered comprehensive, was unsystematic or not clearly reported. The impact of these is difficult to determine.
Additional studies were found by the ERG either by repeating the manufacturers’ searches or conducting independent searches. Reproduction of search strategies also includes identifying additional or new studies that have since been published after submission from the manufacturer. Independent searching by the ERG occurs for two reasons; first, when manufacturer strategies are missing or poorly reported or, second, when there are missing sources, for example, databases of unpublished or ongoing research. It is unclear whether all the studies identified were included in the ERG report and what impact they would have on the outcome of the reports.
Thirteen ERG reports did not critique the manufacturers’ searches for the cost-effectiveness reviews. This may suggest that the manufacturer did not report on the search so the ERG could not appraise it or that the search was adequate. There are only a few limitations that are specific to the cost-effectiveness review searches; it is reasonable to suggest that the criteria that ERGs use to appraise the search methods in the two reviews may be very similar. A majority of the limitations are related to the search strategy and the reporting of the searches. There were fewer weaknesses identified in the sources used for the cost-effectiveness reviews. This could be because of the limited number of economic evaluation sources that manufacturers are able to search by comparison to the clinical effectiveness review.
It is worth commenting that only seven ERG reports mention that the manufacturers’ searches for model parameters were not systematic. The absence of ERG comments on searches in the remaining reports does not necessarily suggest that the manufacturers’ searches were systematic either. To date there is no equivalent guideline on how to systematically search and report on methods for economic evaluation models. According to the NICE guidance, the evidence used for models needs to be systematic, fully documented and justified but there is little guidance on how information is to be identified from sources or how the search should be reported (1;2). Furthermore, there is no checklist in the appraisal of searches for populating models.
Few ERG groups report or acknowledge the use of the PRESS evidence-based checklist (Reference Sampson, McGowan, Lefebvre, Moher and Grimshaw11;Reference McGowan, Sampson and Lefebvre12). There are several reasons for this. First, most of the appraisals were undertaken before the publication of the checklist in 2009. Second, it is not a stated requirement that the ERG should use a checklist in the NICE methods guidance (2). Third, the PRESS checklist although useful for the appraisal of manufacturers’ search strategies was not specifically developed for STAs. For instance, other aspects such as the quality of the reporting, sources used, and techniques applied and comprehensiveness of the searches are appraised by ERGs but not included in the PRESS checklist.
Study Limitations
It was found that different styles were adopted by ERG groups in the reporting of the critique such as layout of the commentary in bullet points or structuring by PICO and more recently, based on the framework provided by the PRESS evidence-based checklist. Some ERGs have provided detailed critiques of individual strategies in the appendices while some have only provided general comments in the main body of the report. There are currently no guidelines for presenting and structuring the critique in the ERG report. The authors did not attempt to study the differences in the styles, structure, layout, and organization within the report in detail.
Even though the literature search process is an integral step in the STA, the impact of the limitations identified by the ERG on the overall findings of the STAs is not clear. Literature surrounding the arguments for and against comprehensive searching in HTAs is still a matter of ongoing debate (Reference Higgins and Green15;Reference Booth23). Studies measuring the impact of errors in search strategies, and the importance of the evidence on precision and recall, have found that where there are mistakes in the search strategies, the effectiveness of the retrieval process is reduced (Reference McGowan, Sampson and Lefebvre12;Reference Royle and Waugh24). However, the impact of any errors is minimized by the STA process because the ERG identifies and, as in the case of missing studies, “corrects” any errors in the searches conducted for a submission. In this sense, the weaknesses of manufacturers’ submission, as judged by ERGs, would not particularly affect decisions because the ERGs not only appraise the submission but currently often also provide the information perceived to be lacking. However, there are obvious benefits for all concerned in the higher quality submissions that might result from manufacturers following the current NICE recommendations and future guidance. First, the ERGs can concentrate on appraising the quality of the searches and find any missed studies necessary for decision making, and second, the appraisal and decision-making process would be enhanced and facilitated by such improvements.
For best practice in STAs, the results from this study show that in addition to the appraisal of the search strategy, the reporting of methods and sources used are equally important by ERG teams. Therefore, the use of the PRESS checklist alone is insufficient to appraise the search methodologies. The appraisal should consist of two key elements: An assessment of the actual reporting of the searches undertaken and a structured critical appraisal of the actual strategy (including source). Manufacturers’ failure to satisfy the first element adequately will obviously adversely affect the second, that is, if the search process and details are poorly reported, then it is likely that the formal appraisal of the actual strategy will also encounter and report problems and limitations. The authors have thus attempted to compile a checklist based on the themes developed from this study in Table 3. The checklist incorporates search strategy, reporting and source elements.
Table 3. Checklist for searches in technology appraisals
Implications for policy and practice
• The findings from this study are applicable to other countries that appraise manufacturer submissions in their health technology assessment process, such as France, Scotland, Australia and Canada.
• Manufacturers and agencies involved in health technology assessments will be aware of potential limitations relating to the search methodology.
• Information specialists will be aware of the types of limitations affecting submissions in health technology appraisals.
• Manufacturers’ will be aware of the types of errors commonly identified in the submissions which may be avoided (e.g., missing additional studies) in future and thus improve the quality of searches. This would improve the efficiency of the STA process.
• It is recommended that ERGs use a checklist incorporating elements in the evaluation of reporting, source and search strategy appraisal of the MS.
Research Recommendations
The value of the checklist developed here should be assessed within HTA programs. A checklist incorporating elements specific for economic evaluation searches such as information used to populate the decision analytic model could also be developed in the future (Reference Kaltenthaler, Tappenden, Paisley and Squires25).
CONCLUSIONS
It is recommended that a checklist that has reporting, source, and search strategy elements be used in the appraisal of the manufacturer's searches during the STA process.
SUPPLEMENTARY MATERIAL
Supplementary Figure 1: www.journals.cambridge.org/thc2013122
CONTACT INFORMATION
Ruth Wong, PhD, (ruth.wong@sheffield.ac.uk), Information Specialist, School of Health and Related Research (ScHARR), University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, United Kingdom.
Suzy Paisley, PhD, Senior Research Fellow, School of Health and Related Research (ScHARR), University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, United Kingdom.
Christopher Carroll, PhD, Senior Lecturer in Health Technology Assessment, School of Health and Related Research (ScHARR), University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, United Kingdom.
CONFLICTS OF INTEREST
Ruth Wong and Suzy Paisley report receiving consultancy money to their institution from BresMed. Christopher Carroll reports grants pending to his institution from NICE, United Kingdom.
Open access
