Uncertainty is a fundamental component of decision making regarding access to and pricing and reimbursement of drugs. The context-specific interpretation and mitigation of uncertainty remain major challenges for decision makers. Following the 2021 HTAi Global Policy Forum, a cross-sectoral, interdisciplinary HTAi-DIA Working Group (WG) was initiated to develop guidance to support stakeholder deliberation on the systematic identification and mitigation of uncertainties in the regulatory-HTA interface.

Six online discussions among WG members (Dec 2021–Sep 2022) who examined the output of a scoping review, two literature-based case studies and a survey; application of the initial guidance to a real-world case study; and two international conference panel discussions.

The WG identified key concepts, clustered into twelve building blocks that were collectively perceived to define uncertainty: “unavailable,” “inaccurate,” “conflicting,” “not understandable,” “random variation,” “information,” “prediction,” “impact,” “risk,” “relevance,” “context,” and “judgment.” These were converted into a checklist to explain and define whether any issue constitutes a decision-relevant uncertainty. A taxonomy of domains in which uncertainty may exist within the regulatory-HTA interface was developed to facilitate categorization. The real-world case study was used to demonstrate how the guidance may facilitate deliberation between stakeholders and where additional guidance development may be needed.

The systematic approach taken for the identification of uncertainties in this guidance has the potential to facilitate understanding of uncertainty and its management across different stakeholders involved in drug development and evaluation. This can improve consistency and transparency throughout decision processes. To further support uncertainty management, linkage to suitable mitigation strategies is necessary.

This study aimed to compare assessments between Beneluxa Initiative member countries’ assessments and identify alignments and divergences.

A retrospective comparative analysis was performed that investigated (i) number and type of assessed indications (for Austria (AT), Belgium (BE), Ireland (IE), and the Netherlands (NL)); (ii) added benefit conclusions (for BE, IE, and NL); and (iii) the main arguments underlying differences in conclusions (for BE, IE, and NL). Data were retrieved directly from agency representatives and from public HTA reports. European Medicines Agency approved indications were included for drugs assessed between 2016 and 2020, excluding veterinary drugs, generics, and biosimilars.

Only 44 (10 percent) of the 444 included indications were assessed by all four member countries. Between any pair of two countries, the overlap was higher, from 63 (AT–NL) to 188 (BE–IE). Added benefit conclusions matched exactly in 62–74 percent of the indications, depending on the countries compared. In the remaining cases, most often a difference of one added benefit level was observed (e.g., higher vs. equal relative effect). Contradictory outcomes were very rare: only three cases were observed (lower vs. higher effect). When assessing the underlying arguments for seven cases with different outcomes, differences were attributable to slight differences in weighing of evidence and uncertainties rather than disagreement on aspects within the assessment itself.

Despite high variability in European HTA procedures, collaboration on HTA between the Beneluxa Initiative member countries is very feasible and would likely not result in added benefit conclusions that would be very different from added benefit conclusions in national procedures.

Conducting a cost-effectiveness analysis (CEA) is resource consuming, and therefore the Dutch National Health Care Institute (ZIN) only performs those for interventions with a high budget impact. Sometimes, cost-effectiveness (CE) estimates are clearly below or far above reference values, which makes full cost-effectiveness assessments less vital. The objective of this study was to develop an efficient and simplified method to identify interventions that are clearly cost-(in)effective.

The method makes use of headroom analysis. Several HTA experts and other relevant stakeholders have been asked to provide feedback on a preliminary version of the CE signal.

The method consists of five steps. In the first step (i) the relevant willingness-to-pay threshold is determined. Reference values are used by ZIN for the maximum willingness-to-pay per incremental quality-adjusted life year (QALY), depending on burden of disease.

In next step (ii) the health gain that can realistically be obtained with the new treatment is estimated. Hereby the effect of the intervention on the clinical outcomes, quality of life and gained life years is determined to estimate the number of QALYs gained, including uncertainty. Then (iii) the societal cost maximum (SCM) of the new treatment is calculated by multiplying step 2 with step 1. In step four (iv) the incremental treatment costs are estimated looking at both the costs and savings for both treatments options for the average patient. In the final step (v) the incremental treatment costs are compared to the SCM to determine if the intervention is probably cost effective, probably not cost-effective or if a conclusion cannot be drawn.

This method has proven to be feasible and could be a valuable addition to the current cost-effectiveness assessment toolbox. The CE-signal is being validated against performing a full cost-effectiveness analysis.

Conducting joint health technology assessments (HTA) is one of the main goals of the Beneluxa Initiative. To strengthen this collaboration, this study aimed to assess similarities and differences between past assessments of Beneluxa Initiative member countries (Austria, Belgium, Ireland and the Netherlands).

A retrospective comparative analysis was performed that investigated the similarities and differences in drug assessments in the period 2016 to 2020 in (i) the number and type of assessed indications; (ii) the conclusions within assessments performed by at least two member countries; and (iii) the main arguments leading to the conclusions through a qualitative analysis of selected cases, looking into the patient population, the intervention, comparator, outcome, timing, and included evidence.

The scope of HTA differs between the countries, with Belgium and Ireland assessing most, the Netherlands focusing on drugs above a budget impact threshold and Austria on outpatient drugs. Furthermore, indications might slightly differ between countries. Therefore, only 44 (10%) of the 444 included drug-indication combinations were assessed through a full HTA by all four countries. Between any pair of countries, the overlap was higher, from 63 (Austria-the Netherlands) to 188 (Belgium-Ireland). Added benefit conclusions matched exactly in 62 to 76 percent of the indications, depending on the compared countries. In the remaining cases, often a difference of one added benefit level was observed (e.g., higher versus equal relative effect). Contradictory outcomes were very rare. Differences were observed with regards to whether a cost-effectiveness analysis was performed. When assessing the underlying arguments within the reports for nine cases with different outcomes, it became clear that organizations agree on almost all aspects, and that differences are mostly attributable to slight differences in weighing of some aspects and uncertainties.

Overall, which indications are assessed differs, but for those indications that are assessed by multiple member countries, considerations and assessment outcomes are similar.

The constant evolution of high-priced hospital drugs with large uncertainties are increasingly challenging the sustainability of many healthcare reimbursement systems. Consequently, more attention is paid to exploring innovative payment and reimbursement models that may contribute to sustainable healthcare funding mechanisms yet still ensure timely patient access to health intervention. However, more insight is needed into the experiences of stakeholders involved in making payment and reimbursement arrangements as they may play an essential part in successfully implementing innovative reimbursement and payment models in the future.

Between May and July 2021, a survey was sent out to Dutch stakeholders directly or indirectly involved in payment and reimbursement agreements. The survey questioned the current use of and future preferences for reimbursement and payment models for high priced hospital drugs using Likert scales. Results were analyzed using descriptive statistics.

Our inquiry shows that currently financial-based reimbursement models are applied most often, especially discounts due to their simplicity. In the future, outcome-based reimbursement models are preferred more, particularly pay-for-outcome. The main challenge for implementation is generating evidence in practice.

Upfront payment is currently the most frequently applied payment model, according to the respondents, whereas delayed payment models are preferred to be applied more often. Particularly payment at outcome-achieved models are preferred but they can be administratively difficult to arrange.

The respondents are moderately satisfied with the payment and reimbursing system in the Netherlands, arguing that especially transparency about the final agreement and mutual trust could be improved.

Despite the preference healthcare payers have for using outcome-based reimbursement models and delayed payment models, currently they are rarely applied. The insights from our survey can provide Dutch stakeholders with future direction when implementing innovative reimbursement and payment models. Attention should be paid to which barriers are currently perceived and how policy structures can facilitate implementation. Further research should focus on exploring which combinations of reimbursement and payment models are most likely to be successful in the Netherlands.

Risk prediction models, using either machine learning or statistical algorithms, can act as inputs of a cost-effectiveness model when predicting costs and effectiveness of an intervention. This systematic review has two objectives: to evaluate methodological quality of the published models to predict diabetic coronary heart disease (CHD) risk; to evaluate whether the models were sufficiently reported to judge their applicability to the cost-effectiveness modelling.

A targeted review of journal articles published in English, Dutch, Chinese, or Spanish was undertaken in PubMed, Embase, Scopus, Web of Science, and IEEE Explore from 1 January, 2016 to 31 May, 2021. To assess the methodological quality and reporting of the models, we used PROBAST (Prediction model Risk Of Bias Assessment Tool), CHARMS (a Checklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies), and a checklist (Betts 2019) summarizing the application of cardiovascular risk prediction models to health technology assessment.

Our search retrieved 6,579 hits, of which 18 models were eligible for inclusion. Among them, four studies developed machine learning models (2 recurrent neural networks, 1 random forest models, and 1 multi-task learning model) while 14 studies developed statistical models (8 Cox models, 5 logistic models, and 1 microsimulation model). More than 70 percent of models were of high methodological quality in aspects of participants (89%), predictors (72%), and outcomes (72%), while only five models (28%) in aspects of statistical analysis. For the reporting, only two models provided sufficient evidence in all aspects (i.e., participants, predictors, and outcomes) for judging their applicability to the cost-effectiveness modelling. Most models were reported sufficiently regarding participants (78%) and outcomes (72%), but only three models regarding predictors (17%).

To apply the CHD risk prediction models to cost-effectiveness modelling, concerns remain regarding the potential risk of bias due to inappropriate use of analysis methods, and regarding insufficient reporting on how to measure and assess the predictors.