Health technologies are becoming increasingly complex and contemporary health technology assessment (HTA) is only partly equipped to address this complexity. The project “Integrated assessments of complex health technologies” (INTEGRATE-HTA), funded by the European Commission, was initiated with the overall objective to develop concepts and methods to enable patient-centered, integrated assessments of the effectiveness, and the economic, social, cultural, and ethical issues of complex technologies that take context and implementation issues into account. The project resulted in a series of guidances that should support the work of HTA scientists and decision makers alike.

Objectives: Current health technology assessment (HTA) is not well equipped to assess complex technologies as insufficient attention is being paid to the diversity in patient characteristics and preferences, context, and implementation. Strategies to integrate these and several other aspects, such as ethical considerations, in a comprehensive assessment are missing. The aim of the European research project INTEGRATE-HTA was to develop a model for an integrated HTA of complex technologies.

Methods: A multi-method, four-stage approach guided the development of the INTEGRATE-HTA Model: (i) definition of the different dimensions of information to be integrated, (ii) literature review of existing methods for integration, (iii) adjustment of concepts and methods for assessing distinct aspects of complex technologies in the frame of an integrated process, and (iv) application of the model in a case study and subsequent revisions.

Results: The INTEGRATE-HTA Model consists of five steps, each involving stakeholders: (i) definition of the technology and the objective of the HTA; (ii) development of a logic model to provide a structured overview of the technology and the system in which it is embedded; (iii) evidence assessment on effectiveness, economic, ethical, legal, and socio-cultural aspects, taking variability of participants, context, implementation issues, and their interactions into account; (iv) populating the logic model with the data generated in step 3; (v) structured process of decision-making.

Conclusions: The INTEGRATE-HTA Model provides a structured process for integrated HTAs of complex technologies. Stakeholder involvement in all steps is essential as a means of ensuring relevance and meaningful interpretation of the evidence.

Objectives: Internationally, funders require stakeholder involvement throughout health technology assessment (HTA). We report successes, challenges, and lessons learned from extensive stakeholder involvement throughout a palliative care case study that demonstrates new concepts and methods for HTA.

Methods: A 5-step “INTEGRATE-HTA Model” developed within the INTEGRATE-HTA project guided the case study. Using convenience or purposive sampling or directly / indirectly identifying and approaching individuals / groups, stakeholders participated in qualitative research or consultation meetings. During scoping, 132 stakeholders, aged ≥ 18 years in seven countries (England, Italy, Germany, The Netherlands, Norway, Lithuania, and Poland), highlighted key issues in palliative care that assisted identification of the intervention and comparator. Subsequently stakeholders in four countries participated in face–face, telephone and / or video Skype meetings to inform evidence collection and / or review assessment results. An applicability assessment to identify contextual and implementation barriers and enablers for the case study findings involved twelve professionals in the three countries. Finally, thirteen stakeholders participated in a mock decision-making meeting in England.

Results: Views about the best methods of stakeholder involvement vary internationally. Stakeholders make valuable contributions in all stages of HTA; assisting decision making about interventions, comparators, research questions; providing evidence and insights into findings, gap analyses and applicability assessments. Key challenges exist regarding inclusivity, time, and resource use.

Conclusion: Stakeholder involvement is feasible and worthwhile throughout HTA, sometimes providing unique insights. Various methods can be used to include stakeholders, although challenges exist. Recognition of stakeholder expertise and further guidance about stakeholder consultation methods is needed.

Objectives: The INTEGRATE-HTA project provided methodology to evaluate complex technologies. This study provides guidance on how to retrieve and critically appraise available evidence on moderators and predictors of treatment effects and on patient preferences for treatment outcomes as a source of complexity.

Methods: Search filters for PubMed were developed by hand-searching a large volume of articles reporting on relevant aspects. Search terms were retrieved from selected papers and algorithmically combined to find the optimal combination of search terms. For the development of the appraisal checklists literature was searched in PubMed and Google Scholar together with citation chasing. For the CHecklist for the Appraisal of Moderators and Predictors (CHAMP) a Delphi procedure was used to value a set of eligible appraisal criteria retrieved from the literature.

Results: Search filters were developed optimized for different accuracy measures. The final version of CHAMP consists of a seventeen questions covering the design, analysis, results and transferability of results of moderator and predictor analysis. The final checklist for appraisal of literature on patient preferences for treatment outcomes consist of six questions meant to help the user to identify relevant quality issues together with a guidance toward existing tools concerning the appraisal of specific preference elicitation methods.

Conclusions: Incorporating knowledge on subgroups for whom a specific treatment will produce more benefit holds the promise of better targeting and, ultimately, enhancing overall effectiveness and efficiency of healthcare technology. Finally, incorporating information on preferences for treatment outcomes will foster health technology assessment that addresses outcomes that are important to patients.

Objectives: Despite recent development of health technology assessment (HTA) methods, there are still methodological gaps for the assessment of complex health technologies. The INTEGRATE-HTA guidance for effectiveness, economic, ethical, socio-cultural, and legal aspects, deals with challenges when assessing complex technologies, such as heterogeneous study designs, multiple stakeholder perspectives, and unpredictable outcomes. The objective of this article is to outline this guidance and describe the added value of integrating these assessment aspects.

Methods: Different methods were used to develop the various parts of the guidance, but all draw on existing, published knowledge and were supported by stakeholder involvement. The guidance was modified after application in a case study and in response to feedback from internal and external reviewers.

Results: The guidance consists of five parts, addressing five core aspects of HTA, all presenting stepwise approaches based on the assessment of complexity, context, and stakeholder involvement. The guidance on effectiveness, health economics and ethics aspects focus on helping users choose appropriate, or further develop, existing methods. The recommendations are based on existing methods’ applicability for dealing with problems arising with complex interventions. The guidance offers new frameworks to identify socio-cultural and legal issues, along with overviews of relevant methods and sources.

Conclusions: The INTEGRATE-HTA guidance outlines a wide range of methods and facilitates appropriate choices among them. The guidance enables understanding of how complexity matters for HTA and brings together assessments from disciplines, such as epidemiology, economics, ethics, law, and social theory. This indicates relevance for a broad range of technologies.

Objectives: The translation of research findings into policy and practice is crucially dependent on the applicability of such findings in a given decision-making context. We explored in a case study whether a generic consultation guide to assess the applicability of a health technology could be rapidly deployed and deliver useful insights.

Methods: A consultation guide based on the context and implementation for complex interventions (CICI) framework was developed and piloted to assess the applicability of reinforced home-based palliative care in three European countries. Individual consultations in England and Germany and a panel discussion in Poland were completed.

Results: Various barriers may hinder successful implementation of reinforced home-based palliative care in the three countries. Whilst the experts across all countries emphasized the lack of funding along with organization and structure as major barriers, information varied by country for many of the other identified barriers and facilitators. Participants in the pilot study provided positive feedback in terms of understanding the topic and purpose of the consultation, and both individual and panel consultations could be easily implemented.

Conclusions: In this case study, the consultation guide presented a pragmatic, ready-to-use tool to assess the applicability of a health technology. As shown here, it can be used in a generic manner without discrete empirical information on the technology in question or, ideally, makes use of specific information collected as part of a HTA. Further studies are needed to validate this guide and apply it to other types of health technologies and more diverse decision-making contexts.

Objectives: A comprehensive health technology assessment (HTA) enables a patient-centered assessment of the effectiveness, economic, ethical, socio-cultural, and legal issues of health technologies that takes context and implementation into account. A question is whether these various pieces of evidence need to be integrated, and if so, how that might be achieved. The objective of our study is to discuss the meaning of integration in the context of HTA and suggest how it may be achieved in a more structured way.

Methods: An analysis of the concept of integration in the context of HTA and a review of approaches that were adopted in the INTEGRATE-HTA project that may support integration.

Results: Current approaches to integration in HTA are mainly methods of commensuration, which are not optimally geared to support public deliberation. In contrast, articulating evaluative frameworks could be an important means of integration which allows for exploring how facts and values can be brought to bear on each other.

Conclusions: Integration is not something that only needs to be addressed at the end, but rather throughout an HTA, right from the start. Integration can be conceived as a matter of accounting for the relevance of empirical evidence in view of a commitment to a set of potentially conflicting values. Various elements of the INTEGRATE-HTA project, such as scoping and the development of logic models, can help to achieve integration in HTA.

Objectives: The aim of this study was to describe an initial exploration by CADTH, Canada's pan-Canadian health technology assessment (HTA) agency, in using the INTEGRATE-HTA guidance in the production of an HTA that examined the use of both in-center and in-home dialysis modalities for the treatment of end-stage kidney disease in adults in Canada.

Methods and Results: We outline CADTH's standard HTA production process and context and then describe the experience of the assessment team in using the INTEGRATE-HTA guidance, specifically to help structure and guide the use of a logic model, the identification of implementation issues, and the identification and examination of ethical issues. For each of the aspects, we describe and reflect on how the assessment team used the guidance, challenges that were encountered in its use, and whether and how we might address these challenges when using the INTEGRATE-HTA guidance in the future.

Conclusions: INTEGRATE-HTA provided detailed and helpful guidance for truly integrating wide-ranging aspects of HTA. Our agency was challenged by a steep learning curve for assessment team members, tight project timelines, and a misalignment of current HTA processes with those required to implement the guidance. Nevertheless, using the guidance initiated a dialogue about what might be needed to assess complex interventions and the potential process changes that could facilitate conducting more integrated assessments.

A central idea underlying the INTEGRATE-HTA project is that many of the interventions that are being used in health care are quite complex. By this, we mean that the relation between the delivery of the intervention on the one hand, and the onset of (desired and undesired) changes may be less straightforward than hoped for. There may be all sorts of reasons for this, varying from a lack of resources, lack of skills, perverse incentives, organizational problems, etc. Not identifying such factors and their potential impact may seriously compromise the policy relevance of a health technology assessment (HTA) (1). However, current approaches and methods in HTA do not seem to be adequately geared to deal with this complexity.

Objectives: The INTEGRATE-HTA project recommends that complexity be taken into account when conducting health technology assessments (HTAs) and suggests a five-step process for doing that. This study examines whether the approach suggested by INTEGRATE-HTA could be useful, appropriate, and feasible in the context of low- and middle-income countries (LMIC) given some of the typical challenges that healthcare systems face in those countries.

Methods: A nonexhaustive literature review was performed on the implementation in low and middle income countries of the five aspects recommended by the INTEGRATE-HTA project, using the following search terms: national health planning, health sector strategy, health sector performance, assessment criteria, health (management) information, complexity, context, stakeholder consultation.

Results: HTA is being practiced in LMIC in various ways and through different mechanisms, for example in health sector reviews, even though it is usually not referred to as HTA. It does not necessarily follow the five steps distinguished in the INTEGRATE-HTA model (scoping; defining the initial logic model; providing concepts and methods to identify, collect, and synthesize evidence in relation to various dimensions; extracting and presenting evidence in respect of agreed assessment criteria; providing guidance to draw conclusions and formulate recommendations).

Conclusions: The conditions for functional HTA are not always fulfilled in LMICs. At least four aspects would require special attention: (a) the scope and quality of routine health information that can support and be fed into health technology assessments and strategic planning; (b) consensus on health system performance assessment frameworks and their main criteria, in particular the inclusion of social disparities/equity and sustainability; (c) institutional capacity to set evidence-based priorities based on a variety of explicit criteria; (d) political will to engage with stakeholders in a transparent and inclusive consultation process about health priorities.

The HTA Core Model (HTACM) of EUnetHTA and the INTEGRATE-HTA Model (IHTAM) both provide HTA experts with advanced guidance on how to assess health technologies. In this study, we examine the similarities and differences of the two models, identifying synergies and opportunities for future collaboration. We also consider how such an alignment of the HTACM and IHTAM might be done in practice and present some alternative practical approaches. Overall the two models share several similarities, primarily the perception of HTA as a multidisciplinary analysis that needs to be adjusted according to the properties of the technology under assessment.