Skip to main content Accessibility help


  • Kei Long Cheung (a1) (a2), Silvia M.A.A. Evers (a1) (a3), Hein de Vries (a4) and Mickaël Hiligsmann (a1)


Objectives: Several studies have reported multiple barriers to and facilitators for the uptake of health technology assessment (HTA) information by policy makers. This study elicited, using best-worst scaling (BWS), the most important barriers and facilitators and their relative weight in the use of HTA by policy makers.

Methods: Two BWS object case surveys (one for barriers, one for facilitators) were conducted among sixteen policy makers and thirty-three HTA experts in the Netherlands. A list of twenty-two barriers and nineteen facilitators was included. In each choice task, participants were asked to choose the most important and the least important barrier/facilitator from a set of five. We used Hierarchical Bayes modeling to generate the mean relative importance score (RIS) for each factor and a subgroup analysis was conducted to assess differences between policy makers and HTA experts.

Results: The five most important barriers (RIS > 6.00) were “no explicit framework for decision-making process,” “insufficient support by stakeholders,” “lack of support,” “limited generalizability,” and “absence of appropriate incentives.” The six most important facilitators were: “availability of explicit framework for decision making,” “sufficient support by stakeholders,” “appropriate incentives,” “sufficient quality,” “sufficient awareness,” and “sufficient support within the organization.” Overall, perceptions did not differ markedly between policy makers and HTA experts.

Conclusions: Our study suggests that barriers and facilitators related to “policy characteristics” and “organization and resources” were particularly important. It is important to stimulate a pulse at the national level to create an explicit framework for including HTA in the decision-making context.



Hide All
1. Park, AL, McDaid, D, Weiser, P, et al. Examining the cost effectiveness of interventions to promote the physical health of people with mental health problems: A systematic review. BMC Public Health. 2013;13:787. PubMed PMID: 23988266. Pubmed Central PMCID: 3765875.
2. Nicod, E, Kanavos, P. Commonalities and differences in HTA outcomes: A comparative analysis of five countries and implications for coverage decisions. Health Policy. 2012;108:167177.
3. Drummond, MF, Schwartz, JS, Jönsson, B, et al. Key principles for the improved conduct of health technology assessments for resource allocation decisions. Int J Technol Assess Health Care. 2008;24:244258.
4. van Velden, ME, Severens, JL, Novak, A. Economic evaluations of healthcare programmes and decision making. Pharmacoeconomics. 2005;23:10751082.
5. Oliver, K, Innvar, S, Lorenc, T, et al. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2. PubMed PMID: 24383766. Pubmed Central PMCID: 3909454.
6. Macintyre, S, Chalmers, I, Horton, R, et al. Using evidence to inform health policy: Case study. BMJ. 2001;322:222.
7. Garrido, MV. Health technology assessment and health policy-making in Europe: Current status, challenges and potential. Denmark: WHO Regional Office Europe; 2008.
8. Drummond, M. Economic evaluation in health care: Is it really useful or are we just kidding ourselves? Aust Econ Rev. 2004;37:311.
9. Neumann, PJ. Why don't Americans use cost-effectiveness analysis. Am J Manag Care. 2004;10:308312.
10. Neumann, PJ, Sullivan, SD. Economic Evaluation in the US. Pharmacoeconomics. 2006;24:11631168.
11. Prosser, LA, Koplan, JP, Neumann, PJ, et al. Barriers to using cost-effectiveness analysis in managed care decision making. Am J Manag Care. 2000;6:173179.
12. Mühlbacher, AC, Kaczynski, A, Zweifel, P, et al. Experimental measurement of preferences in health and healthcare using best-worst scaling: An overview. Health Econ Rev. 2015;6 (1):114.
13. Cheung, KL, Wijnen, BF, Hollin, IL, et al. Using best–worst scaling to investigate preferences in health care. Pharmacoeconomics. 2016;34:11951209.
14. Finn, A, Louviere, JJ. Determining the appropriate response to evidence of public concern: The case of food safety. J Public Policy Mark. 1992:1225.
15. Train, KE. Discrete choice methods with simulation. Cambridge, UK: Cambridge University Press; 2009.
16. Flynn, TN, Louviere, JJ, Peters, TJ, et al. Best–worst scaling: What it can do for health care research and how to do it. J Health Econ. 2007;26:171189.
17. Marley, AA, Louviere, JJ. Some probabilistic models of best, worst, and best–worst choices. J Math Psychol. 2005;49:464480.
18. Merlo, G, Page, K, Ratcliffe, J, et al. Bridging the gap: Exploring the barriers to using economic evidence in healthcare decision making and strategies for improving uptake. Appl Health Econ Health Policy. 2014;13:303309.
19. van Gool, MK, Gallego, G, Haas, M, et al. Economic evidence at the local level. Pharmacoeconomics. 2007;25:10551062.
20. Williams, I, Bryan, S. Understanding the limited impact of economic evaluation in health care resource allocation: A conceptual framework. Health Policy. 2007;80:135143.
21. Huić, M, Nachtnebel, A, Zechmeister, I, et al. Collaboration In health technology assessment (EU net HTA joint action, 2010–2012): Four case studies. Int J Technol Assess Health Care. 2013;29:323330.
22. Drummond, M, Weatherly, H. Implementing the findings of health technology assessments. Int J Technol Assess Health Care. 2000;16:112.
23. Hivon, M, Lehoux, P, Denis, J-L, et al. Use of health technology assessment in decision making: Coresponsibility of users and producers? Int J Technol Assess Health Care. 2005;21:268275.
24. Stakes, H, Rius, ME, Espinas, JA. EUR-ASSESS project subgroup report on dissemination and impact. Int J Technol Assess Health Care. 1997;13:220286.
25. Brousselle, A, Lessard, C. Economic evaluation to inform health care decision-making: Promise, pitfalls and a proposal for an alternative path. Soc Sci Med. 2011;72:832839.
26. Hoffmann, C, Stoykova, BA, Nixon, J, et al. Do health‐care decision makers find economic evaluations useful? The findings of focus group research in UK health authorities. Value Health. 2002;5:7178.
27. Johnson, RM. Understanding HB: An intuitive approach. Sequim, WA: Sawtooth Software Inc; 2000.
28. Orme, B. Hierarchical Bayes: Why all the attention? Quirk's Mark Res Rev. 2000;14:1663.
29.Sawtooth Software. Identifying ‘bad’ respondents: Fit Statistic and Identifying Random Responders. 2016. (accessed January 27, 2016).
30. Ehlers, L, Jensen, MB. Attitudes and barriers toward mini-HTA in the Danish municipalities. Int J Technol Assess Health Care. 2012;28:271277.
31. Oortwijn, W, Broos, P, Vondeling, H, et al. Mapping of health technology assessment in selected countries. Int J Technol Assess Health Care. 2013;29:424434.
32. Cheung, KL, Evers, SM, Hiligsmann, M, et al. Understanding the stakeholders’ intention to use economic decision-support tools: A cross-sectional study with the Tobacco Return on Investment tool. Health Policy. 2016;120:4654.
33. de Vries, H, Mudde, A, Leijs, I, et al. The European Smoking prevention Framework Approach (EFSA): An example of integral prevention. Health Educ Res. 2003;18:611626.
34. de Vries, H, Eggers, SM, Bolman, C. The role of action planning and plan enactment for smoking cessation. BMC Public Health. 2013;13: 393.
35. De Vries, H, Eggers, SM, Jinabhai, C, et al. Adolescents’ beliefs about forced sex in KwaZulu-Natal, South Africa. Arch Sex Behav. 2014;43: 19.
36. Hoffmann, C, von der Schulenburg, J-MG. The influence of economic evaluation studies on decision making.: A European survey. Health Policy. 2000;52:179192.


Type Description Title
Supplementary materials

Cheung supplementary material
Table S1

 Word (16 KB)
16 KB
Supplementary materials

Cheung supplementary material
Table S2

 Word (15 KB)
15 KB
Supplementary materials

Cheung supplementary material
Table S3

 Word (15 KB)
15 KB


  • Kei Long Cheung (a1) (a2), Silvia M.A.A. Evers (a1) (a3), Hein de Vries (a4) and Mickaël Hiligsmann (a1)


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed