Skip to main content Accessibility help

Health Technology Agency insights: informing modification of a qualitative benefit risk framework for Health Technology Reassessment of prescription medications

  • Mary Alison Maloney (a1) (a2), Lisa Schwartz (a1), Daria O'Reilly (a3) (a4) and Mitchel Levine (a3)



This study's intent was to determine if a qualitative benefit risk framework could be used or modified to further enable Health Technology Reassessment (HTR) of prescription medicine recommendations. The purpose of this research was to understand Canadian Health Technology Agency assessors past experiences and insights to inform any modifications to the Universal Methodology for Benefit−Risk Assessment (UMBRA) qualitative framework. The UMBRA framework consists of an eight-step process, used during the assessment phase, to aid in decision making and dissemination.


A qualitative descriptive study was conducted and included a purposeful, criterion-based sample of eight assessors who had participated in Health Technology Assessment (HTA) or HTR for prescription medicines or in qualitative decision-making frameworks.


Participant interviews lead to four common themes: “adoption of a qualitative benefit risk framework,” “data (either too much or not enough),” “importance of incorporating stakeholder values,” and “feasibility of the UMBRA framework.” Methodological challenges with HTR were highlighted including the lack of clinical outcome data and the ability to compare clinically relevant meaningful differences. The implementation of a ranking or weighing process found within the UMBRA framework was not favored by half of the participants.


Research participants did not consider all steps of the UMBRA framework to be transferable to the assessment phase of HTR given the need for simplicity, resource efficiency, and stakeholder input throughout the process. The assessor experiences and insights and the resultant key themes can be used in future research to aid in the development of a qualitative recommendation framework for HTR.


Corresponding author

Author for correspondence: Mary Alison Maloney, E-mail: or


Hide All
1.Hughes, D, Waddingham, EAD, Mt-Isa, S et al. (2013) Recommendations for the methodology and visualisation techniques to be used in the assessment of benefit and risk of medicines. Available at: Accessed October 2016.
2.Phillips, LD, Fasolo, B, Zafiropoulos, N et al. (2010) Benefit-risk methodology project. Work package 2 report: Applicability of current tools and processes for regulatory benefit-risk assessment. 31 August 2010. Available at: Accessed October 2016.
3.Leong, J, Salek, S, Walker, S (2014) Strategy for communicating benefit-risk decisions: a comparison of regulatory agencies' publicly available documents. Front Pharmacol. 5(269), 19.
4.Mt-Isa, S, Wang, N, Hallgreen, CE et al. (2013) Review of methodologies for benefit and risk assessment of medication. Available at: Accessed October 2016.
5.Mt-Isa, S, Hallgreen, CE, Wang, N et al. (2014) Balancing benefit and risk of medicines: a systematic review and classification of available methodologies. Pharmacoepidemiol Drug Saf 23(7), 667678.
6.Mt-Isa, S, Ouwens, B, Robert, V et al. (2016) Structured benefit-risk assessment: a review of key publications and initiatives on frameworks and methodologies. Pharma Stat. 15, 4.
7.Pignatti, F, Ashby, D, Brass, EP et al. (2015) Structured frameworks to increase the transparency of the assessment of benefits and risks of medicines: current status and possible future direction. Clin Pharmacol Ther. 98(5), 522533.
8.European Medicines Agency (2012) Benefit-risk methodology project. Work package 4 report: benefit-risk tools and processes. Available at: Accessed October 2016.
9.US Food and Drug Administration (2018) Benefit-risk assessment in drug regulatory decision-making. Draft PDUFA VI implementation plan (FY 2018–2022). March 2018. Available at: Accessed October 2018.
10.Walker, S, McAuslane, N, Liberti, L, Leong, J, Salek, S (2015) A universal framework for the benefit-risk assessment of medicines: is this the way forward? Ther Innov Regul Sci. 49(1), 1725.
11.Centre for Innovation in Regulatory Science (2016) UMBRA Initiative. Available at: Accessed October 2016.
12.MacKean, G, Noseworthy, T, Elshaug, AG et al. (2013) Health technology reassessment: the art of the possible. Int J Technol Assess Health Care. 29, 418423.
13.Maloney, MA, Schwartz, L, O'Reilly, D, Levine, M (2017) Drug disinvestment frameworks: components, challenges and solutions. Int J Technol Assess Health Care. 33, 261269.
14.Seo, HJ, Park, JJ, Lee, SH (2016) A systematic review on current status of health technology reassessment: insights for South Korea. Health Res Policy Sys. 14, 110.
15.Soril, LJ, MacKean, G, Noseworthy, TW, Leggett, LE, Clement, FM (2017) Achieving optimal technology use: a proposed model for health technology reassessment. SAGE Open Med. 5, 17.
16.Canadian Agency for Drugs and Technologies in Health (2016) Therapeutic Review Framework and Process. Aug 2016 Version 2.5. Available at: Accessed June 2016.
17.Centre for Innovation in Regulatory Science (2013) Is there a commonality across the structured decision frameworks used by HTA and regulatory agencies? Workshop report. October 2013. Available at: Accessed July 2018.
18.Sullivan-Bolyai, S (2005) Developing and refining interventions in persons with health disparities: the use of qualitative description. Nurs Outlook. 53, 127133.
19.Neergaard, MA, Olesen, F, Andersen, RS, Sondergaard, J (2009) Qualitative description—the poor cousin of health research? BMC Med Res Methodol. 9, 5257.
20.Sandelowski, M (2000) Whatever happened to qualitative description? Res Nurs Health. 23, 334340.
21.Creswell, JW (2013) Qualitative inquiry and research design: choosing among five traditions. 3rd ed. Thousand Oaks: Sage Publications.
22.Miles, MB, Huberman, AM (1994) Qualitative data analysis—an expanded sourcebook. 2nd ed. Thousand Oaks: Sage Publications.
23.Smith, JA (2007) Qualitative psychology: a practical guide to research methods. 2nd ed. London: Sage Publications.
24.Glaser, BG (1965) The constant comparative method of qualitative analysis. Soc Probl. 12(4), 436445.
25.Fram, S (2013) The constant comparative analysis method outside of grounded theory. Qual Rep. 18(1), 125.
26.Emden, C, Sandelowski, M (1998) The good, the bad, and the relative, part one: conceptions of goodness in qualitative research. Int J Nurs Pract. 4, 206212.
27.Whittemore, R, Chase, S, Mandle, C (2001) Validity in qualitative research. Qual Health Res. 11, 522537.
28.Milne, J, Oberle, K (2005) Enhancing rigor in qualitative description: a case study. J Wound Ostomy Continence Nurs. 32, 413420.
29.Canadian Agency for Drugs and Technologies in Health (2016) Recommendation framework for CADTH Common Drug Review and pan-Canadian Oncology Drug Review programs: Guidance for CADTH's drug expert committees. March 2016. Available at: Accessed June 2016.
30.Canadian Agency for Drugs and Technologies in Health (2016) pCODR expert review committee deliberative framework. August 2016. Available at: Accessed June 2016.
31.DECIDE GRADE (2017) DECIDE 2011–2015. Available at: Accessed June 2017.
32.Elshaug, A, Watt, A, Moss, J, Hiller, J (2009) Policy perspectives on the obsolescence of health technologies in Canada. Available at: Accessed 6 February 2016.
33.Henshall, C, Schuller, T, Mardhani-Bayne, L (2012) Using health technology assessment to support optimal use of technologies in current practice: the challenge of “disinvestment”. Int J Technol Assess Health Care. 28, 203210.
34.Esmail, R, Hanson, H, Holroyd-Leduc, J, Niven, DJ, Clement, F (2018) Knowledge translation and health technology reassessment: identifying synergy. BMC Health Serv Res. 18(1), 674682.
35.Giorgi, A (1992) Description versus interpretation: competing alternative strategies for qualitative research. J Phenomenol Psychol. 23, 119135. 10.1163/156916292X00090.


Type Description Title
Supplementary materials

Maloney et al. supplementary material
Tables S1-S2

 Word (20 KB)
20 KB


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed