Skip to main content Accessibility help


  • Anna Nachtnebel (a1), Johanna Breuer (a2), Wolfgang Willenbacher (a3), Anna Bucsics (a4), Peter Krippl (a5) and Claudia Wild (a2)...


Objectives: The regularly structured adaptation of health technology assessment (HTA) programs is of utmost importance to sustain the relevance of the products for stakeholders and to justify investment of scarce financial resources. This study describes internal adjustments and external measures taken to ensure the Horizon Scanning Programme in Oncology (HSO) is current.

Methods: Formal evaluation methods comprising a survey, a download, an environmental analysis, and a Web site questionnaire were used to evaluate user satisfaction.

Results: The evaluation showed that users were satisfied with HSO outputs in terms of timeliness, topics selected, and depth of information provided. Discussion of these findings with an expert panel led to changes such as an improved dissemination strategy and the introduction of an additional output, that is, the publication of a league table of emerging oncology drugs. The rather high level of international usage and the environmental analysis highlighted a considerable overlap in topics assessed and, thus, the potential for international collaboration. As a consequence, thirteen reports were jointly published based on eleven “calls for collaboration.” To further facilitate collaboration and the usability of reports for other agencies, HSO reports will be adjusted according to tools developed at a European level.

Conclusions: Evaluation of the impact of HTA programs allows the tailoring of outputs to fit the needs of the target population. However, within a fast developing HTA community, estimates of impact will increasingly be determined by international collaborative efforts. Refined methods and a broader definition of impact are needed to ultimately capture the efficiency of national HTA programs.



Hide All
1. Nachtnebel, A, Geiger-Gritsch, S, Hintringer, K, Wild, C. Scanning the horizon: Development and implementation of an early awareness system for anticancer drugs in Austria. Health Policy. 2012;104:111.
2. Ludwig Boltzmann Institute for Health Technology Assessment. Horizon scanning in oncology. (accessed August 16, 2015).
3. Zechmeister, I, Schumacher, I. The impact of health technology assessment reports on decision making in Austria. Int J Technol Assess Health Care. 2012;28:7784.
4. Douw, K, Vondeling, H. Selection of new health technologies for assessment aimed at informing decision making: A survey among horizon scanning systems. Int J Technol Assess Health Care. 2006;22:177183.
5. Packer, C, Fung, M, Stevens, A. Analyzing 10 years of early awareness and alert activity in the United kingdom. Int J Technol Assess Health Care. 2012;28:308314.
6. Packer, C, Gutierrez-Ibarluzea, I, Simpson, S. The evolution of early awareness and alert methods and systems. Int J Technol Assess Health Care. 2012;28:199200.
7. EuroScan International Network. A toolkit for the identification and assessment of new and emerging health technologies. Birmingham: EuroScan International Network; 2014.
8. Integrate-HTA. (accessed August 10, 2015).
9. Shaping European Early Dialogues for health technologies. (accessed August 10, 2015).
10. Advance-HTA. (accessed August 10, 2015).
11. Gerhardus, A, Dorendorf, E, Rottingen, J, Santamera, A. Health technology assessment and health policy making in Europe - Current status, challenges and potential. Brussels: European Observatory on Health Systems and Policy; 2008.
12. Sheehan, KB. E-mail survey response rates: A review. J Comput Mediat Commun. 2001;6.
13. Zakosteletzki, L, Nachtnebel, A. Horizon scanning in oncology - Impact evaluation and environmental analysis. Wien, Austria: Ludwig Boltzmann Institute for Health Technology Assessment; 2012.
14. Hailey, D, Babidge, W, Cameron, A, Lise, A. HTA agencies and decision makers - An INAHTA guidance document. Alberta, Canada: INAHTA; 2010.
15. Wanke, M, Juzwishin, D, Thornley, R, Chan, L. An exploratory review of evaluations of health technology assessment agencies. HTA Initiative. Edmonton, Alberta: Alberta Heritage Foundation for Medical Research; 2006.
16. Murphy, K, Packer, C, Stevens, A, Simpson, S. Effective early warning systems for new and emerging health technologies: Developing an evaluation framework and an assessment of current systems. Int J Technol Assess Health Care. 2007;23:324330.
17. Raftery, J, Hanney, S, Green, C, Buxton, M. Assessing the impact of England's National Health Service R&D Health Technology Assessment program using the “payback” approach. Int J Technol Assess Health Care. 2009;25:15.
18. American Association for Public Opinion Research Response Rates - An Overview. (accessed January 18, 2015).
19. Hofmarcher, M. Health systems in transition - Austria. Brussels: European Observatory on Health Systems and Policy; 2013.
20. Huic, M, Nachtnebel, A, Zechmeister, I, et al. Collaboration in health technology assessment (EUnetHTA joint action, 2010–2012): Four case studies. Int J Technol Assess Health Care. 2013;29:323330.
21. EUnetHTA. Work package 5 joint action: Relative effectiveness assessment of pharmaceuticals. HTA Core Model® for rapid relative effectiveness assessment of pharmaceuticals. Copenhagen: EUnetHTA; 2013.
22. Hailey, D. Elements of effectiveness for health technology assessment programs. HTA Initiative. Edmonton, Alberta: Alberta Heritage Foundation for Medical Research; 2003.


Type Description Title
Supplementary materials

Nachtnebel supplementary material
Table S1

 Word (30 KB)
30 KB


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed