Health technology assessment (HTA) is the systematic evaluation of various properties and effects of a health technology. HTA can serve as a bridge between the world of knowledge and that of decision making, offering decision makers the best summary of scientific evidence. Scoping HTA reports in the context of dentistry can help researchers identify grey areas; help practitioners make evidence-based decisions and further initiate better policy making.

To provide an overview on HTAs pertaining to oral health and dentistry in the past decade, map the extension and scope of the methodological practices, key findings, and limitations.

A scoping review was conducted using the Joanna Briggs Institute framework. A comprehensive search for HTA reports was done through the International Network of Agencies for Health Technology Assessment Database from January 2010 to December 2020. Consecutively, electronic databases (PubMed and Google Scholar) were searched. Finally, thirty-six reports were included in this review and analyzed.

A total of 709 articles were initially identified, of which thirty-six met the inclusion criteria. Reviewed HTAs focused on various specialties of dentistry worldwide. Maximum number of reports (N = 5) were related to “prosthodontics and dental implants” and technologies related to preventive dentistry were most commonly assessed (N = 4).

Functional, appropriate, and evidence-based information provided through HTA pertaining to oral health on a regular basis will enable decision makers to have enough data to make decisions on the future use of new technology, modify existing policies, accelerate its translation into practice, and ensure provision of robust dental healthcare services.

The hazard ratio (HR) is a commonly used summary statistic when comparing time to event (TTE) data between trial arms, but assumes the presence of proportional hazards (PH). Non-proportional hazards (NPH) are increasingly common in NICE technology appraisals (TAs) due to an abundance of novel cancer treatments, which have differing mechanisms of action compared with traditional chemotherapies. The goal of this study is to understand how pharmaceutical companies, evidence review groups (ERGs) and appraisal committees (ACs) test for PH and report clinical effectiveness in the context of NPH.

A thematic analysis of NICE TAs concerning novel cancer treatments published between 1 January 2020 and 31 December 2021 was undertaken. Data on PH testing and clinical effectiveness reporting for overall survival (OS) and progression-free survival (PFS) were obtained from company submissions, ERG reports, and final appraisal determinations (FADs).

NPH were present for OS or PFS in 28/40 appraisals, with log-cumulative hazard plots the most common testing methodology (40/40), supplemented by Schoenfeld residuals (20/40) and/or other statistical methods (6/40). In the context of NPH, the HR was ubiquitously reported by companies, inconsistently critiqued by ERGs (10/28), and commonly reported in FADs (23/28).

There is inconsistency in PH testing methodology used in TAs. ERGs are inconsistent in critiquing use of the HR in the context of NPH, and even when critiqued it remains a commonly reported outcome measure in FADs. Other measures of clinical effectiveness should be considered, along with guidance on clinical effectiveness reporting when NPH are present.

To identify which international health technology assessment (HTA) agencies are undertaking evaluations of medical tests, summarize commonalities and differences in methodological approach, and highlight examples of good practice.

A methodological review incorporating: systematic identification of HTA guidance documents mentioning evaluation of tests; identification of key contributing organizations and abstraction of approaches to all essential HTA steps; summary of similarities and differences between organizations; and identification of important emergent themes which define the current state of the art and frontiers where further development is needed.

Seven key organizations were identified from 216 screened. The main themes were: elucidation of claims of test benefits; attitude to direct and indirect evidence of clinical effectiveness (including evidence linkage); searching; quality assessment; and health economic evaluation. With the exception of dealing with test accuracy data, approaches were largely based on general approaches to HTA with few test-specific modifications. Elucidation of test claims and attitude to direct and indirect evidence are where we identified the biggest dissimilarities in approach.

There is consensus on some aspects of HTA of tests, such as dealing with test accuracy, and examples of good practice which HTA organizations new to test evaluation can emulate. The focus on test accuracy contrasts with universal acknowledgment that it is not a sufficient evidence base for test evaluation. There are frontiers where methodological development is urgently required, notably integrating direct and indirect evidence and standardizing approaches to evidence linkage.

Treatment seeking for gender dysphoria (GD) has increased manifold in western countries. This has led to increased interest on evidence-base of treatments, but also discussions related to human rights, identity politics, gender-related structures, and medicalization. Combining these discourses into coherent health policy is difficult. Health technology assessment (HTA) is the golden standard for assessing whether a medical intervention should be included in a health system. A comprehensive HTA should include medical, safety, and cost-utility perspectives, but often also ethical, societal, organizational, and legal concerns. Still, ethics is often omitted in practice. This paper aims to demonstrate how integrated ethical analysis influenced a HTA of complex and controversial topics like GD.

A HTA of medical treatments of GD was conducted using integrated ethical analysis based on the EUnetHTA-model. This integrates ethical thinking into the whole HTA, explicitly analyses ethical topics, and balances arguments using several ethical theories.

Integrating ethics had a significant impact on the HTA process and recommendations. It influenced how the HTA was planned and executed, emphasized autonomy and justice when creating the recommendations, and helped the workgroup to understand the complexity of combining different stakeholders’ discourses. Tensions between scientific evidence, expectations, and values became explicit.

Comprehensive HTA provides an important, integrative approach to considering complex and controversial topics in health systems. HTA emphasizes multidisciplinary and multi-stakeholder approach but simultaneously forces a pragmatic, results-oriented, and evidence-based approach on all argumentation. Ethical analysis can facilitate interactions between stakeholders, bridge different discourses, and help formulate widely acceptable guidelines and policy decisions.

Artificial intelligence (AI) is seen as a major disrupting force in the future healthcare system. However, the assessment of the value of AI technologies is still unclear. Therefore, a multidisciplinary group of experts and patients developed a Model for ASsessing the value of AI (MAS-AI) in medical imaging. Medical imaging is chosen due to the maturity of AI in this area, ensuring a robust evidence-based model.

MAS-AI was developed in three phases. First, a literature review of existing guides, evaluations, and assessments of the value of AI in the field of medical imaging. Next, we interviewed leading researchers in AI in Denmark. The third phase consisted of two workshops where decision makers, patient organizations, and researchers discussed crucial topics for evaluating AI. The multidisciplinary team revised the model between workshops according to comments.

The MAS-AI guideline consists of two steps covering nine domains and five process factors supporting the assessment. Step 1 contains a description of patients, how the AI model was developed, and initial ethical and legal considerations. In step 2, a multidisciplinary assessment of outcomes of the AI application is done for the five remaining domains: safety, clinical aspects, economics, organizational aspects, and patient aspects.

We have developed an health technology assessment-based framework to support the introduction of AI technologies into healthcare in medical imaging. It is essential to ensure informed and valid decisions regarding the adoption of AI with a structured process and tool. MAS-AI can help support decision making and provide greater transparency for all parties.

There has been a lack of health technology assessment (HTA) methods for novel digital health technologies (DHTs) such as mHealth, artificial intelligence, and robotics in Finland. The Digi-HTA method has been developed for this purpose. The aim of this study is to determine whether it would be possible to use Digi-HTA recommendations to support healthcare decision-makers. Secondly, from the perspective of companies offering different types of DHT products, this study assesses the suitability of using the Digi-HTA framework to perform HTAs for their products.

Feedback about Digi-HTA recommendations was collected from healthcare professionals. DHT companies provided input about the Digi-HTA framework. Data were collected via a web-based survey and were analyzed using qualitative methods.

Of the twenty-four healthcare professional respondents, twenty said that the Digi-HTA recommendations contained all the necessary information, and twenty-one found them useful for their work. Respondents hoped that the Digi-HTA recommendations would be better integrated into the decision-making processes and healthcare professionals would be more informed about this new HTA process. The questions of the Digi-HTA framework were applicable for different DHT products based on the responses from DHT companies (n = 8).

According to the study participants, although the Digi-HTA recommendations include clear and beneficial information, their integration into healthcare decision-making processes should be improved. Responses from DHT companies indicate that the Digi-HTA framework would be an appropriate tool for performing assessments for their products. To generalize the findings of this study, more comprehensive studies will be needed.

The reimbursement process for innovative health technologies in Hungary lacks any formalized assessment of clinical added benefit (CAB). The aim of this research is to present the development, retrospective testing, and implementation of a local assessment framework for determining the CAB of cancer treatments at the Department of Health Technology Assessment of the National Institute of Pharmacy and Nutrition in Hungary.

The assessment framework was drafted after screening existing methods and a retrospective comparison of local reimbursement dossiers to that of German and French methods. The Magnitude of Clinical Benefit Scale of the European Society for Medical Oncology was chosen to rate the extent of CAB in oncology, as part of a conclusion complemented by the assessment of endpoint relevance and the quality of evidence. Several rounds of retrospective assessments have been conducted involving all clinical assessors, iterated with semistructured discussions to consolidate divergence between assessors. External stakeholders were consulted to provide feedback on the framework.

Retrospective assessments resulted in average more than 75 percent concordance between assessors on each element of the conclusion. Input from ten stakeholders was also incorporated; stakeholders were generally supportive, and they mostly commented on the concept, the elements of the framework, and its implementation.

The procedure is suitable for routine use in the decision-making process to describe the CAB of antineoplastic technologies in Hungary. Further extension of the framework is required to cover more disease areas for structured and comparable conclusions on CAB of innovative health technologies.

To investigate the test–retest reliability of the Costs for Patients Questionnaire (CoPaQ).

Through an online survey, individuals were invited to participate in a two-step study to assess the test–retest reliability of the CoPaQ. Participants to the first step were invited to complete the questionnaire a second time 2 weeks after. Reliability was assessed by calculating Cohen’s Kappa coefficients and intraclass correlation coefficients (ICC) for discrete and continuous data, respectively. A sensitivity analysis was carried out.

From a total of 1,200 participants who completed the first test, 403 completed the second test. The ICC varied from −0.00 to 0.98 with poor, moderate, good, and excellent results. The Kappa coefficients varied from −0.004 to 0.65 and were poor, slight, fair, moderate, and substantial. The sensitivity analysis showed the median value of ICC and Kappa coefficients for each category before and after the outliers’ exclusion. The median value of ICC changed from 0.30 (before) to 0.70 (after), and from 0.12 (before) to 0.04 (after), respectively, for each category. The median value of the Cohen’s Kappa coefficient increased from 0.44 (before) to 0.46 (after) and decreased from 0.32 (before) to 0.30 (after), respectively.

Test–retest reliability results indicated that the CoPaQ has a moderate reliability in terms of ICC and Kappa coefficients. The moderate reliability observed gives additional support for the applicability of this tool in economic evaluations of health interventions. Additional studies including on other properties and a cultural adaptation could further enhance the use of the tool.

Publication bias has the potential to adversely impact clinical decision making and patient health if alternative decisions would have been made had there been complete publication of evidence.

The objective of our analysis was to determine if earlier publication of the complete evidence on rosiglitazone’s risk of myocardial infarction (MI) would have changed clinical decision making at an earlier point in time. We tested several methods for adjustment of publication bias to assess the impact of potential time delays to identifying the MI effect. We then performed a cumulative meta-analysis (CMA) for both published studies (published-only data set) and all studies performed (comprehensive data set). We then created an adjusted data set using existing methods of adjustment for publication bias (Harbord regression, Peter’s regression, and the nonparametric trim and fill method) applied to the limited data set. Finally, we compared the time to the decision threshold for each data set using CMA.

Although published-only and comprehensive data sets did not provide notably different final summary estimates [OR = 1.4 (95 percent confidence interval [CI]: .95–2.05) and 1.42 (95 percent CI: 1.03–1.97)], the comprehensive data set reached the decision threshold 36 months earlier than the published-only data set. All three adjustment methods tested did not show a differential time to decision threshold versus the published-only data set.

Complete access to studies capturing MI risk for rosiglitazone would have led to the evidence reaching a clinically meaningful decision threshold 3 years earlier.

To conduct a formative evaluation of applying the VALIDATE approach in practice by (i) assessing how students appreciated the e-learning course, (ii) exploring how, for what purposes and with what outcomes the acquired VALIDATE competences subsequently were used in internships in different institutional contexts, and how this was shaped by these contexts, and (iii) what this shows on real-world use of VALIDATE.

Comparative discussion of experiences of applying the VALIDATE approach via a semistructured survey and oral feedback from e-course students; final reports on internships in health technology assessment (HTA) practice, followed by semistructured interviews with interns and supervisors to complement and interpret results.

All students considered the VALIDATE approach an enlightening and important addition to current HTA knowledge, especially regarding understanding the relation between empirical analysis and normative inquiry, identifying policy relevant questions and using the method of reconstructing interpretive frames for scoping. The latter appeared intellectually challenging and requiring some prior HTA knowledge. The use the VALIDATE approach in practice shows that interns productively redefined the HTA problem, based on appreciation of different stakeholders’ definition of the issue; they experienced constraints from retrieving all relevant perspectives from existing literature as well as from institutional rules and routines.

Some challenges in applying the VALIDATE approach deserve attention for its future use: currently used research approaches often assume a problem as “given”; and the data needed on different perspectives is often not reported in scientific literature. Finally, data gathering on and evaluation of value dimensions was experienced as challenging.

Argentina has a fragmented healthcare system with social security covering almost two thirds of the population. Its benefit package—called compulsory medical program (PMO; by its Spanish acronym Programa Médico Obligatorio)—has not been formally and widely updated since 2005. However, laws, clinical practice guidelines (CPGs), and a high-cost technology reimbursement fund complement it. Our objective was to comprehensively review such a PMO and propose an update considering the corresponding complementary sources.

We followed four steps: (i) identification of health technologies from the current PMO and complementary sources, (ii) prioritization, (iii) assessment through rapid health technology assessment (HTA), and (iv) appraisal and recommendations. We evaluated three value domains: quality of evidence, net benefit, and economics, which were summarized in a five-category recommendation traffic-light scale ranging from a strong recommendation in favor of inclusion to a strong recommendation for exclusion.

Eight hundred fifty technologies were identified; 164 of those, considered as high priority, were assessed through rapid HTAs. Those technologies mentioned in laws and CPGs were mostly outpatient essential medicines, whereas those from the reimbursement system were mostly high-cost drugs; of these 101 technologies, 50 percent were recommended to be kept in the PMO. The other 63 (identified by the Superintendence of Health Services, technology producers, and patients) were mostly medical procedures and high-cost drugs; only 25 percent of those resulted in a favorable recommendation.

A methodology based on four clearly identified steps was used to carry out a comprehensive review of an outdated and fragmented benefit package. The use of rapid HTAs and a traffic-light recommendation framework facilitated the deliberative evidence-based update.

While ethics has been identified as a core component of health technology assessment (HTA), there are few examples of practical, systematic inclusion of ethics analysis in HTA. Some attribute the scarcity of ethics analysis in HTA to debates about appropriate methodology and the need for ethics frameworks that are relevant to local social values. The “South African Values and Ethics for Universal Health Coverage” (SAVE-UHC) project models an approach that countries can use to develop HTA ethics frameworks that are specific to their national contexts.

The SAVE-UHC approach consisted of two phases. In Phase I, the research team convened and facilitated a national multistakeholder working group to develop a provisional ethics framework through a collaborative, engagement-driven process. In Phase II, the research team refined the model framework by piloting it through three simulated HTA appraisal committee meetings. Each simulated committee reviewed two case studies of sample health interventions: opioid substitution therapy and either a novel contraceptive implant or seasonal influenza immunization for children under five.

The methodology was fit-for-purpose, resulting in a context-specified ethics framework and producing relevant findings to inform application of the framework for the given HTA context.

The SAVE-UHC approach provides a model for developing, piloting, and refining an ethics framework for health priority-setting that is responsive to national social values. This approach also helps identify key facilitators and challenges for integrating ethics analysis into HTA processes.

In spring 2020, The European network for Health Technology Assessment (EUnetHTA) decided to join forces to produce best evidence to inform health policy in the COVID-19 pandemic. The objective of this paper is to describe the process and output of the coordinated and collaborative activities of EUnetHTA.

Relevant published and internal documents were retrieved for a descriptive analysis of EUnetHTA processes, methods, and outputs related to EUnetHTA’s response to the pandemic.

Process: In April 2020, a COVID-19 task force was set up and a survey collected pressing health policy questions across Europe. Two coordinating agencies for diagnostic tests and therapeutics were assigned. A process for prioritization and selection was set up for therapeutics, as well as explicit starting and stopping rules. Methodology: To increase a timely response, it was agreed that the rapid collaborative reviews (rapid CRs) would not require the consultation of manufacturers and the involvement of external experts, but would not differ in the methods and conduct of the systematic search, review, and synthesis of all available evidence, nor in the requirement for reviewing by EUnetHTA partners. Final reports: The joint effort resulted in the production of two rapid CRs on diagnostic tests, nineteen collaborative rolling reviews on therapeutics, three of which later moved to rapid CRs.

During COVID-19 pandemic, the EUnetHTA partners proved capable of prompt collaboration, which allowed speeding up the production and release of high-quality EUnetHTA outputs, while the relationships with the other European institutions facilitated their quick dissemination.

In model-based economic evaluations, the effectiveness parameter is often informed by studies with a limited duration of follow-up, requiring extrapolation of the treatment effect over a longer time horizon. Extrapolation from short-term data alone may not adequately capture uncertainty in that extrapolation. This study aimed to use structured expert elicitation to quantify uncertainty associated with extrapolation of the treatment effect observed in a clinical trial.

A structured expert elicitation exercise was conducted for an applied study of a podiatry intervention designed to reduce the rate of falls and fractures in the elderly. A bespoke web application was used to elicit experts’ beliefs about two outcomes (rate of falls and odds of fracture) as probability distributions (priors), for two treatment options (intervention and treatment as usual) at multiple time points. These priors were used to derive the temporal change in the treatment effect of the intervention, to extrapolate outcomes observed in a trial. The results were compared with extrapolation without experts’ priors.

The study recruited thirty-eight experts (geriatricians, general practitioners, physiotherapists, nurses, and academics) from England and Wales. The majority of experts (32/38) believed that the treatment effect would depreciate over time and expressed greater uncertainty than that extrapolated from a trial-based outcome alone. The between-expert variation in predicted outcomes was relatively small.

This study suggests that uncertainty in extrapolation can be informed using structured expert elicitation methods. Using structured elicitation to attach values to complex parameters requires key assumptions and simplifications to be considered.

Presenting an approach to synthesize quantitative and qualitative information from systematic reviews of multiple health interventions.

Within the context of an EUnetHTA multi-health technology assessment of twenty-three surgical techniques, we developed synthetic single tables, using color gradients and abbreviations, with information on which technologies had been compared, estimates of the size of differences for available comparisons, their clinical relevance, and certainty of the related evidence.

The proposed methodology provided, through a single depiction, information normally included in multiple figures/tables such as network plots, league tables, and summary of findings tables.

Transferring information on benefits, risks, and certainty of the available evidence on health interventions may be challenging, especially when assessing multiple treatments: more pieces of information need to be integrated in order to show an overall picture for each of the chosen outcomes, and usual reporting tools may be targeted to researchers more than to different kinds of decision makers. While more in-depth layers of information can always be added to satisfy needs of different audiences, the proposed tools could favor a quick interpretation of articulated scientific data by both decision makers and researchers.

Patient and public involvement (PPI) in the Brazilian Health Technology Assessment (HTA) process occurs in response to a legislative mandate for “social participation.” This resulted in some limited patient participation activities, and, therefore, a more systematic approach was needed. The study describes the development of a suggested framework for action to improve PPI in HTA.

This work used formal methodology to develop a PPI framework based on three-phase mixed-methods research with desktop review of Brazilian PPI activities in HTA; workshop, survey, and interviews with Brazilian stakeholders; and a rapid review of international practices to enact effective patient involvement. Patient partners reviewed the draft framework.

According to patient group representatives, their involvement in the Brazilian HTA process is important but could be improved. Different stakeholders perceived barriers, identified values, and made suggestions for improvement, such as expansion of communication, capacity building, and transparency, to support more meaningful patient involvement. The international practices identified opportunities for earlier, more active, and collaborative PPI during all HTA stages, based on values and principles that are relevant for Brazilian patients and the public. These findings were synthesized to design a framework that defines and systematizes actions to support PPI in Brazil, highlighting the importance of evaluating these strategies.

Since the publication of this framework, some of its suggestions are being implemented in the Brazilian HTA process to improve PPI. We encourage other HTA organizations to consider a systematic and planned approach with regular evaluation when pursuing or strengthening involvement practices.

Mini health technology assessment (HTA) reports have been used to support policy makers and health systems by providing a timely summary of scientific evidence. The objective of this meta-epidemiologic study was to evaluate the quality of reporting of mini-HTA reports published in Brazil.

An electronic search for all mini-HTA reports published between 2014 and March 2019 was conducted in the SISREBRATS and CONITEC databases. The study selection and data extraction were performed by two independent assessors. The following data were extracted: bibliographic data; research question; characteristics of the population, health technologies and outcomes assessed; eligibility criteria; information about searches and study selection; risk of bias assessment; quality of evidence assessment; synthesis of results; and recommendation about the technology evaluated. A descriptive analysis was used to summarize the information retrieved from all the included mini-HTA reports.

We included 103 mini-HTA reports, the great majority of which (92.3 percent) focused on the coverage of the technologies in the healthcare system, with more than 60 percent being about drugs. Only five mini-HTA reports (4.8 percent) gave reasons for the choice of outcomes, and fifteen (14.5 percent) discriminated between primary and secondary outcomes. All mini-HTAs reported the databases searched and 99 percent of them reported using Medline. Sixty percent of the mini-HTA reported assessing the risk of bias, and 52 percent reported assessing the quality of evidence.

The quality of reporting of the mini-HTA reports performed in Brazil is insufficient and needs to be improved to guarantee transparency and replicability.

In 2019, members of the Health Technology Assessment international (HTAi) Interest Group for Disinvestment and Early Awareness (DEA-IG) and the HTAi Interest Group for Information Retrieval (IR-IG) agreed to produce quarterly current awareness alerts for members of the DEA-IG. The purpose was to pilot a predefined strategy for sharing new publications on methods and topical issues in this area.

Literature search strategies for PubMed and Google were developed. Retrieved citations were posted on the DEA-IG Web site. Members of the DEA-IG received an email notification when new alerts were available. An informal survey of the DEA-IG members was used to provide feedback after the pilot.

Six alerts were issued during the pilot (June 2019–September 2020) with a total of 170 citations. The bulk of the information were 124 PubMed indexed citations, and of these, 96 were retrieved by the PubMed search strategies. Google searches were not found to be useful, but ongoing horizon scanning work at the Canadian Agency for Drugs and Technologies in Health (CADTH) provided additional information. Based on retrospective sorting, we considered thirty-five PubMed citations to be highly relevant for health technology assessment (HTA). The response rate to the survey was limited (seventeen respondents), but most respondents found the alerts useful for their work.

The results of this pilot project can be used to revise search strategies and information sources, improve the relevance of the alerts, and plan for expanded dissemination strategies.

We evaluated the early impact of a new hospital-based health technology assessment (HB-HTA) program, called Smart Innovation, at the University of Washington Medical Center (UWMC).

We compared the UWMC's utilization trends for two surgical procedures to control hospitals by evaluating the difference before and after adoption decisions: (i) a new filter for transcatheter aortic valve replacement (TAVR) procedures that treat aortic valve stenosis and (ii) microwave ablation (MWA) for treating hepatocellular carcinoma. We used descriptive statistics to assess the difference between the UWMC and controls for TAVR and MWA procedures and multivariate difference-in-differences (DID) analyses to test for statistical significance.

The UWMC experienced a 10 percent reduction in TAVR procedures compared with controls following the implementation of the TAVR Sentinel filter. The DID regression model indicated a 1.5 reduction in the number of TAVR procedures per quarter at the UWMC between the pre- and post period, which was not statistically significant (p-value: .87). The UWMC experienced a 51 percent reduction in utilization when compared with controls for MWA procedures in the pre- and post periods. The DID model for MWA indicated an 18.8 decrease in utilization per quarter during the study period for the UWMC, which was statistically significant (p-value: .0007). For MWA procedures, the UWMC experienced a $647,658 dollar reduction in total costs in the post period compared with controls.

When the UWMC used HB-HTA methods for technology adoption, there was a reduction in utilization and total costs when compared with controls; however, when the UWMC adopted a new technology without using HB-HTA methods, there was no difference in utilization.

Peer review of searches is a process whereby both the search strategies and the search process description are reviewed, ideally using an evidence-based checklist.

As the search strategy underpins any well-conducted evidence synthesis, its quality could affect the final result. Evidence shows, however, that search strategies are prone to error.

There is increasing awareness and use of the PRESS Evidence-Based Checklist and peer review of search strategies, at the outset of evidence syntheses, prior to the searches being run, and this is now recommended by a number of evidence synthesis organizations.

Searches for evidence syntheses should be peer reviewed by a suitably qualified and experienced librarian or information specialist after being designed, ideally, by another suitably qualified and experienced librarian or information specialist. Peer review of searches should take place at two important stages in the evidence synthesis process; at the outset of the project prior to the searches being run and at the prepublication stage. There is little empirical evidence, however, to support the effectiveness of peer review of searches. Further research is required to assess this. Those wishing to stay up to date with the latest developments in information retrieval, including peer review of searches, should consult the SuRe Info resource (http://www.sure-info.org), which seeks to help information specialists and others by providing easy access to the findings from current information retrieval methods research and thus support more research-based information retrieval practice.