Skip to main content Accessibility help
Hostname: page-component-747cfc64b6-db5sh Total loading time: 0.319 Render date: 2021-06-17T03:22:34.494Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": true, "newCiteModal": false, "newCitedByModal": true, "newEcommerce": true }


Published online by Cambridge University Press:  10 July 2018

Murray Krahn
Toronto General Hospital Research Institute, Toronto Health Economics and Technology Assessment Collaborative (THETA)
Fiona Miller
University of Toronto, Institute of Health Policy, Management and Evaluation
Ahmed Bayoumi
University of Toronto Institute of Health Policy Management and Evaluation Li Ka Shing Knowledge Institute, Centre for Research on Inner City Health, The Keenan Research Centre
Ann-Sylvia Brooker
No affiliation at present
Frank Wagner
University of Toronto Joint Centre for Bioethics
Shawn Winsor
University of Toronto Dalla Lana School of Public Health
Mita Giacomini
McMaster University, Clinical Epidemiology & Biostatistics
Ron Goeree
No affiliation at present
Holger Schünemann
McMaster University, Departments of Clinical Epidemiology and Biostatistics and of Medicine
Gabrielle van der Velde
University of Toronto, Institute of Health Policy, Management and Evaluation
Stephen Petersen
Health Quality Ontario, Evidence Development and Standards
Nancy Sikich
Health Quality Ontario, Evidence Development and Standards
Irfan Dhalla
Health Quality Ontario, Evidence Development and Standards
Rights & Permissions[Opens in a new window]



In 2007, the Ontario Health Technology Advisory Committee (OHTAC) developed a decision framework to guide decision making around nondrug health technologies. In 2012, OHTAC commissioned a revision of this framework to enhance its usability and deepen its conceptual and theoretical foundations.


The committee overseeing this work used several methods: (a) a priori consensus on guiding principles, (b) a scoping review of decision attributes and processes used globally in health technology assessment (HTA), (c) presentations by methods experts and members of review committees, and (d) committee deliberations over a period of 3 years.


The committee adopted a multi-criteria decision-making approach, but rejected the formal use of multi-criteria decision analysis. Three broad categories of attributes were identified: (I) context criteria attributes included factors such as stakeholders, adoption pressures from neighboring jurisdictions, and potential conflicts of interest; (II) primary appraisal criteria attributes included (i) benefits and harms, (ii) economics, and (iii) patient-centered care; (III) feasibility criteria attributes included budget impact and organizational feasibility.


The revised Ontario Decision Framework is similar in some respects to frameworks used in HTA worldwide. Its distinctive characteristics are that: it is based on an explicit set of social values; HTA paradigms (evidence based medicine, economics, and bioethics/social science) are used to aggregate decision attributes; and that it is rooted in a theoretical framework of optimal decision making, rather than one related to broad social goals, such as health or welfare maximization.

Copyright © Cambridge University Press 2018 

In 2003, the Ministry of Health and Long-Term Care (MOHLTC) in Ontario, Canada's largest province (population 13.6 million), recognized the need for a system to address the uptake and diffusion of nondrug health technologies. The Ontario Health Technology Advisory Committee (OHTAC) was created to serve as a single provincial portal for recommendations on the introduction of nondrug health technologies (Reference Johnson, Sikich and Evans1).

Health technology assessment (HTA) includes several phases: (a) prioritization of the health technology; (b) assessment of relevant evidence; (c) deliberation and recommendation development; and (d) dissemination and elicitation of stakeholder views (2).

Within OHTAC, methods for systematically considering all of the relevant types of evidence in the appraisal process (phase III) have continued to evolve. In January 2007, OHTAC developed a decision-making framework that included four key attributes: (a) overall clinical benefit, (b) consistency with expected societal and ethical values, (c) value for money, and (d) feasibility (system feasibility and economic feasibility) of adoption into the health system (Reference Johnson, Sikich and Evans1). This framework offered a transparent, multi-disciplinary and consistent approach to making decisions in a deliberative manner. The framework recognized the importance of different forms of evidence, and included ethical considerations and social values (Reference Johnson, Sikich and Evans1). OHTAC's approach has been adapted and used by other HTA groups (3;4).

Several operational and conceptual issues prompted a re-evaluation of the framework. Assigning clinical evidence into ordinal categories (e.g., strength of evidence: high versus medium versus low) proved to be difficult in practice due to lack of specific guidance. Systematic collection of evidence related to societal values and ethics was not routinely undertaken; instead colloquial evidence from available experts was used. Feasibility considerations were usually not supported by direct evidence.

Conceptually, the growing importance of patient perspectives and the international prominence of other decision and value frameworks also highlighted the need to reconsider the conceptual foundations of our work. Decision frameworks rooted in clinical evidence (e.g., GRADE) (Reference Alonso-Coello, Oxman and Moberg5) are being widely used in clinical, public health, and policy contexts. Multi-criteria decision frameworks such as EVIDEM (Reference Goetghebeur, Wagner and Khoury6) are increasingly being explored. Value frameworks promulgated by clinical specialty groups (Reference Anderson, Heidenreich and Barnett7;Reference Schnipper, Davidson and Wollins8) are receiving increasing attention, particularly in the United States. Each of these approaches includes a (often implicit) theoretical framework, a set of decision attributes, and a method to integrate them into decision making.

Therefore, OHTAC requested a re-evaluation of our decision-making process. In January 2012, OHTAC convened two subcommittees: (a) a Public Engagement Subcommittee, which began the process of developing a patient and public engagement strategy (Reference Abelson, Wagner and DeJean9); and (b) a Decision Determinants (DD) Subcommittee to develop a decision-making framework, called the “Decision Determinants Framework.”


Decision Determinants Subcommittee Objectives

The objectives of the DD Subcommittee were to (a) critically review OHTAC's existing decision determinants framework (2007), and (b) provide advice on potential theoretical and practical improvements.

OHTAC members and relevant experts from bioethics, evidence based medicine, clinical medicine, social science, and health economics, were brought together to provide consensus recommendations for an update of the current decision framework. The committee met monthly for a period of 3 years. Our methods included: (a) Consensus on a set of guiding principles; (b) A scoping literature review, including a review of the gray literature and Web sites of international HTA organizations to determine a comprehensive set of decision attributes used in HTA decision making; (c) Solicited presentations from experts in HTA from a variety of disciplines, such as evidence-based medicine, health economics, decision analysis, bioethics, and health-policy; and (d) Deliberations at monthly meetings between March 2012 and December 2015.

Guiding Principles

A Values-Based Framework

The DD Subcommittee took the view that alignment with social values represents the foundational principle for decision making regarding health technology. The committee believed that all decision criteria, and all decision processes used in the HTA process should be rooted in values about health, and more specifically, about the use of technology. The perspective that social values play a primary role in decision making is consonant with the views of other HTA bodies, which recognize the foundational role of social values (10Reference Whitty and Littlejohns12).

An inquiry into social values commissioned by OHTAC was taken as a starting point to define the social values that are relevant to the work of the committee (Figure 1) (Reference Giacomini, Wagner and Krahn13;Reference Miller14). Giacomini generated a list of consensus values that included: 1. quality, a meta-concept referring to the achievement of the good and continuous striving for improvement; 2. evidence-informed policy; 3. effectiveness; 4. resource stewardship; 5. resource sufficiency; 6. equity; 7. solidarity; 8. population health; 9. patient-centered care; 10. collaboration; and 11. shared responsibility for health. Values 1–5 are more explicitly linked to traditional HTA practice. These values are discussed further in the Scientific Paradigms section.

Figure 1. Social values relevant to the work of Ontario Health Technology Advisory Committee (OHTAC). EUnetHTA, European Network for Health Technology Assessment; HTA, health technology assessment; IJTAHC, International Journal of Technology Assessment in Health Care; INAHTA, International Nerwork of Agencies for Health Technology Assessment; MOHLTC, Ministry of Health and Long-Term Care.


Supplementary Table 1 identifies over 150 health concepts relevant to decisions about health technology. However, research into human cognition has shown that it is difficult to keep more than five to seven concepts in mind at any time (Reference Miller15;Reference Redelmeier, Rozin and Kahneman16). We, therefore, believed that our framework was likely to be useful in guiding decision making to the extent that it reduced the number of factors considered in decision making.

A Guide to Deliberation, not a Substitute for Deliberation

Deliberation, in the context of HTA, refers to a type of discussion in which there is a careful weighing of reasons for and against some proposition (Reference Abelson, Forest and Eyles17). Collective problem-solving is the critical element of deliberation in which individuals from different backgrounds and experiences are given the opportunity to listen, understand, and potentially persuade. This may lead to more reasoned, informed, and socially legitimate decisions (Reference Stafinski, Menon and Marshall11). We believed that the deliberative process was central to decision making that was fair, wise, reproducible, and grounded in public values.

Decision Attributes, Decision Rules, and Process

Decision frameworks include three components: (a) attributes that define the scope of decision making; (b) decision rules for those attributes, which may be formal and explicit, as in the case of multi-criteria decision analysis, or less formal; (c) the process by which decisions are made. Process includes components, such as the composition of the committee; how information is gathered, synthesized and presented, and by whom; how the deliberation is structured; and how recommendations are formulated and disseminated. Though we understood our remit to include all three components, recommendations about process will be reported elsewhere.

Scientific Paradigms – From Values to Methods

Although there is little literature describing the theoretical foundation of health technology, it does draw from a range of disciplines including evidence based medicine, health economics, bioethics and the interpretive social sciences.

The DD Subcommittee explicitly recognized that these different disciplines differ in terms of the questions they ask, the kind of empirical data they seek, the analytic techniques they use, the academic background and training of researchers within these domains, and the social institutions that disseminate their ideas. For example, within evidence-based medicine: (a) practitioners are often clinicians; (b) clinical outcomes (e.g., death, myocardial infarction) are privileged; and (c) epidemiological methods, such as randomized trials, are regarded as the most reliable source of data. Within health economics: (a) practitioners are often social scientists; (b) cost and preferences for health outcomes, in addition to clinical outcomes are privileged; and (c) model based analyses that integrate varied types of data are regarded as the optimal method to support decision making. In bioethics and social science: (a) practitioners are often philosophers or social scientists; (b) patient experience, power relationships, and ethical principles are privileged; (c) methods include ethical reasoning and qualitative (and occasionally quantitative) studies of experiences, values, and preferences.

We suggest that each of the three major “paradigms” that are attached to HTA differ in key areas: “what is to be observed and scrutinized, the kind of questions that are supposed to be asked, how these questions are to be structured, and how the results of scientific investigations should be interpreted.” These are four key features of paradigms, according to Thomas Kuhn, the philosopher of science with whom the term “paradigm” is most associated. Kuhn says that a paradigm is “an entire constellation of beliefs, values and techniques…shared by the members of a given community” (Reference Kuhn18).

The committee recognized that each of these “scientific paradigms” (Reference Kuhn18;Reference Dogan, Smelser and Baltes19) provide independent, valuable, and complementary ways of thinking about health and health technology. Each has developed rigorous methods to evaluate some component of health technology. Most importantly, however, each of these paradigms can be linked to one or more fundamental social value.

Evidence based medicine is linked most closely to values of effectiveness, evidence informed policy, and quality. Health economics is linked to resource stewardship, resource sufficiency, and population health. By explicitly representing the relationship between constrained resources and health, health economics highlights the importance of making decisions such that the overall population can benefit. Social science/bioethics is most clearly linked to values of equity, solidarity, and patient-centered care. This paradigm reflects the importance of fairness in both process and outcome.

The subcommittee wanted to ensure that these paradigms were appropriately represented within the revised framework. Thus, we divided the work of the committee into three working groups based on their shared epistemological perspective and methodological approach.


A scoping review was conducted to identify an exhaustive list of decision attributes, concepts, or ideas used in HTA decisions. We identified 1–12 sub-attributes or related health concepts for each of the twenty-eight high-level attributes, for a total of 157 health concepts.


In the first year, the committee heard a wide variety of presentations from key informants with expertise in the areas of decision analysis, health economics, evidence-based medicine, bioethics, and health policy. Each presented their experience with a specific decision-making framework, including EVIDEM and other multi-criteria decision frameworks (Reference Goetghebeur, Wagner and Khoury6), GRADE (Reference Alonso-Coello, Oxman and Moberg5), and frameworks used by the Pan-Canadian Oncology Drug Review and Ontario Committee to Evaluate Drugs.


The literature review and the solicited presentations provided an opportunity for the subcommittee to discuss how other frameworks might be adapted. Between March 2012 and December 2015, monthly meetings were held by the DD Subcommittee. In addition, the working groups conducted further literature searches into their sub-domains and held their own deliberations. The working groups were responsible for assembling a list of relevant criteria within their domain, as well as describing a suitable methodological approach for generating judgments using those criteria. Each working group also pilot-tested the decision criteria and presented their work at monthly committee meetings.


Multi-criteria Decision Making, but not Multi-criteria Decision Analysis (MCDA)

There was little conflict around the idea that the framework should incorporate discrete decision criteria or attributes. Thus, the committee was committed to a multi-criteria decision-making approach, in which the attributes relevant to decision making serve as the organizing principle for decision making (Reference Devlin and Sussex20). However, the committee did not endorse an MCDA approach, in which explicit attribute weights and decision rules are used to determine the optimal choice.

Some members noted that explicit weights can standardize the decision-making process and potentially reduce bias. Although use of weights can be formulaic and/or prescriptive, in practice, it need not be. Instead, it can support decision making by stimulating reflection and exchange and by making the decision process more explicit. This approach may also mitigate the effects of influential committee members. However, the consensus was that structuring decision making in this way introduces a degree of rigidity into the process that was, on balance, undesirable. One concern was that the score would produce a strong “framing” effect. Another related to independence: in formal MCDA models, criteria should be independent. Some members expressed the view that a quantitative tool may result in placing undue weight on the elements of HTA that are more easily measured. Patient experiences and broader considerations of social value might not receive appropriate consideration and decision weight.

Opportunity Cost in Health and Other Domains

The issue of opportunity cost was vigorously, even hotly debated. The conflict generated by this discussion led to the resignation of several members of the committee. Several of the health economists on the committee believed that opportunity cost was such a central concept that it superseded the role of other decision attributes. They argued that any decision to adopt a particular technology imposed an opportunity cost associated with technologies or interventions that were foregone or displaced. This opportunity cost is usually represented as health foregone, but in a multi-criteria decision framework, opportunity cost might also fall on other domains, such as benefits and risk, or equity. Thus, they argued, the implications of a choice (opportunity cost) were fundamental considerations in arriving at any decision rule, irrespective of which attributes were used.

Other members, while acknowledging the centrality of opportunity cost, also recognized that determining where opportunity cost falls in the Canadian health context is extremely difficult. The extent to which costs imposed on the health budget displaces health, or fall on other budgets is unclear. The committee could not identify any method of operationalizing the concept of opportunity cost in other domains. The committee recognized the pioneering work being done in the United Kingdom to establish an empirical foundation for an estimate of opportunity cost within the health system (Reference Claxton, Martin and Soares21), and strongly endorsed the need for similar work to be done in Canada.

Overall Outline of the Decision Framework

The committee considered that the instrument should have three main categories: (I) context criteria, (II) appraisal criteria, and (III) feasibility criteria.

I. Context Criteria

The committee recognized the importance of framing or contextual considerations in evaluating new technology (Table 1). Understanding who has requested the review and why (i.e., stakeholder issues and pressures), the nature of the technology, and the patients and populations who might benefit, are important. Coverage decisions and availability of the technology in neighboring jurisdictions were also seen as important, as were the scope of the technology appraisal and the affiliation of those who conducted it. The committee did not believe that context criteria should be considered to be primary appraisal criteria.

Table 1. Contextual Factors and Health System Feasibility Components of the Revised Framework

II. Appraisal Criteria

i. Benefits and Harms

The name of the domain was changed to “Benefits and Harms” from “Clinical Evidence,” because the latter term suggested a focus on strictly biomedical interventions and excluded items such as public health interventions (Table 2). The framework now refers to “harms” rather than to “safety.” The subcommittee believed that changing this terminology captures a broader range of types of health interventions and studies.

Table 2. Benefits and Harms Component of the Revised Framework

QALY, quality-adjusted life-year.

Magnitude and Certainty of Benefit and Harm

The new framework explicitly considers both the magnitude of benefit and the certainty regarding this benefit. Magnitude refers to the size of health benefit (e.g. absolute or relative risk reduction, quality-adjusted life-year [QALY] gain). Certainty refers to confidence that benefit will actually be achieved. Uncertainty may be statistical (or random) uncertainty (quantified with p-values or confidence intervals) or bias (systematic uncertainty). The new framework similarly considers the magnitude and certainty of harm.

The selection of the most appropriate measure of benefit should be ascertained at the time of evidence review and will ideally be guided by the availability of data, the opinions of experts, and input from patients (see below).

Representation of estimates of the certainty of benefit and the certainty of harm are related to the GRADE assessment of overall certainty. GRADE assesses uncertainty by explicitly considering both the precision of the estimate and potential biases (Reference Keeney22).

Patients’ Perspectives

The new framework incorporates patients’ perspectives in the Benefits and Harms domain. A frequent criticism of outcomes reported in RCTs is that they may not fully reflect the priorities and values of people who are directly affected (Reference Schoemaker23;Reference von Neumann, Morgenstern and Kuhn24). Patient input is needed to put such outcomes into perspective, particularly when nonmortality related outcomes are being considered, or when the minimum clinically important difference of an outcome measure has not been established.

The inclusion of “patient perspectives” in the Benefits and Harms domain raises questions of double counting (see the Patient Centered Care Domain section, below). This issue was discussed at length. Our final recommendation was that patient perspectives be included here because: (a) the focus on patient perspectives of Benefits and Harms recognizes that such assessments require value judgements, for which patients are perhaps the most authoritative source of information; and (b) patient input is gathered routinely for every assessment, whereas a more detailed assessment is recommended only for selected technologies.

ii. Economics
Cost-Effectiveness Likelihood

The committee suggested that five categories of cost-effectiveness likelihood be implemented as follows (Table 3): (a) highly likely to be cost-effective (80–100 percent), (b) moderately likely to be cost-effective (60–79 percent), (c) uncertain cost-effectiveness (40–59 percent), (d) moderately likely to not be cost-effective (20–39 percent), and (e) highly likely to not be cost-effective (0–19 percent).

Table 3. Economics and Patient-Centered Care Components of the Revised Framework

CE, cost-effectiveness; QALY, quality-adjusted life-year.

Cost-Effectiveness Threshold Values

OHTAC does not have a threshold for determining cost-effectiveness. Until an empirical threshold can be determined, the committee proposed the use of two illustrative ICER thresholds: (a) $50 K/QALY, and (b) $100 K/QALY.

Adequacy of Cost and Outcome Measures

Recognizing that the cost and outcome measures used to derive cost and effect estimates may not capture all relevant information, may be of poorer quality or may have varying levels of relevancy to the Ontario context, the next component of the framework consists of grading of the perceived adequacy of the cost and outcomes measures included in the economic assessment. Specifically, the revised framework includes an input for the user to determine the: (a) comprehensiveness and validity of the cost and outcome valuation/ aggregation; and (b) appropriateness of the cost and outcome measures for Ontario.

Comprehensiveness refers to the completeness with which resource usage estimation was conducted (e.g., were important costs omitted?), or the extent to which important health outcomes were captured (were important health effects omitted?). Validity may refer to either the quality of the decision model, or the appropriateness of the QALY measure in the decision context. For example, the QALY model may not be congruent with social values in circumstances such as end of life care, or “lifestyle” treatments.

Appropriateness refers to the generalizability of the analysis to the Ontario context, including particularly resource use and valuation, and patterns of care.

Summary of Cost-Effectiveness Likelihood

As a final step, the framework combines the quantitative cost-effectiveness results with assessment of comprehensiveness and appropriateness to provide a summary measure for the economic domain.

The summary measure is identical to the previously-mentioned categories but includes a possible “down-grading” of the economic rating if cost, outcomes, or the model itself are considered to be inadequate.

iii. Patient-Centered Care

The previous category of “Social and Ethical Values” was rechristened as “Patient-Centered Care” (Table 3). This new characterization was partly strategic, but it was also recognition of the primacy of the patient's experience in making health decisions.

The Patient-Centered Care subcommittee identified a set of five values of particular salience to OHTAC's commitment to ethics and social values. These include “equity,” “solidarity,” “population health,” “collaboration,” and “shared responsibility for health” (Reference Anderson, Heidenreich and Barnett7).

Trigger Tool

The subcommittee developed a “trigger tool,” adapted from the European Network for Health Technology Assessment (EUnetHTA) model, to determine when a full ethics and social values analysis is warranted. We chose to develop a trigger tool because resources are not available at HQO for full assessments for all technologies. Selected trigger components include: (a) technology decision is identified as being value (or preference) sensitive; (b) use by vulnerable or marginalized populations; and (c) screening interventions; (d) autonomy, privacy, or confidentiality issues especially salient.

With respect to methods, the subcommittee recommended the use of scoping reviews, using a variety of methods. These include: (a) qualitative research synthesis; (b) research synthesis related to health equity; (c) research synthesis for ethics studies; and (d) research synthesis for quantitative evidence of patient preferences (25).

Based on the results of the scoping review, the issues related to alignment with patients’ values and preferences, autonomy, privacy and confidentiality, equity in access or outcomes, and coordination of care (Table 3) should be identified. As with the other domains one of five ordinal categories (strongly supports to strongly discourages use/adoption of the technology) will be selected, based on the body of preference and values evidence identified.

III. Feasibility Criteria

In addition to scientific evidence, the committee considers aggregate budget impact and organizational considerations (Table 1). These include considerations related to impact on human resources (availability of expertise, impacts on work and income of existing personnel) as well as factors related to the organization and administration of health care.


Significance of Adopting a Multi-Criteria Framework

In one sense, the idea that health decisions with complicated medical, moral, economic, and organizational consequences involve the consideration of multiple criteria seems too obvious to mention, let alone belabor. It is difficult to identify any decision, in fact, from the most trivial consumer decision, to broad questions affecting societies or the environment, that do not involve considerations of multiple attributes. This was the key insight behind Keeney's extension of expected utility theory to multi-attribute utility theory (Reference Keeney22). What is not so generally appreciated, though, are the potential implications of characterizing social decision making in this way. Thinking about what it is that a society values, and seeking to structure decisions and decision making in a way that is consonant with those values focuses attention on the fact that decisions, and decision making are at the heart of HTA. The theoretical underpinnings of this approach are related to normative decision theories, ideas about how optimal decisions can be made (Reference Keeney22Reference von Neumann, Morgenstern and Kuhn24).

This is distinct from theoretical perspectives shaped by explicit social objectives. For example, the objective of the National Institute for Health and Care Excellence is the maximization of health, subject to the budget constraint of a fixed NHS budget (25). The theoretical underpinnings of this approach are economic theories of the social good, such as welfarism or extrawelfarism (Reference Brouwer, Culyer and van Exel26). This approach privileges outcomes (societal welfare, health); ours privileges decisions.

Both approaches (making good decisions, and maximizing health or welfare) seem intuitively attractive. And indeed, these approaches should be broadly consonant, if the things we value in decision making are also the social goals our decision making is designed to optimize. Yet, there are important practical consequences of choosing one approach over the other. Seeing decision making about health as a “health maximization problem” places health economic modeling at the center of decision making. Clinical evidence, patient preferences and social values are also relegated to a secondary role. Our committee took the view that a multi-criteria approach is preferable, because it allows each attribute, and the underlying social values to be understood through the prism of the intellectual traditions that have evolved over time to study them.

Significance of a Framework Based on Values

HTA agencies often refer to broad ethical principles which serve as the foundation of their decision making. For example, autonomy, nonmaleficence, beneficence, and distributive justice have been cited as key ethical principles that underlie social value judgments (10). But it is hard to see how clinical evidence review, or cost effectiveness, the practical operational work of HTA, are related to these rather lofty goals. Our work in identifying a broader set of fundamental social values makes it possible to be more explicit about how practical work is linked to values.

Significance of a Framework Based on “Paradigms”

Our work also recognizes that there are intellectual traditions that have developed ways of framing questions, conducting studies, and interpreting results in ways that are distinct and valuable. Our framework organizes attributes by shared epistemological and disciplinary perspectives. For example, some health economists have advocated addressing equity concerns with the use of equity weights in cost effectiveness analysis. The committee's view was that methods and perspectives of interpretive social scientists and bioethicists would likely give equity considerations deeper consideration and greater weight. Similarly, the “health economics paradigm” considers cost and effectiveness together, and, therefore, so do we. In contrast, some frameworks (Reference Goetghebeur, Wagner and Khoury6) explicitly disaggregate these attributes because of concerns related to double counting. The committee's view was that health economics has developed a set of well worked out methods for thinking about value, which would be largely lost if these attributes were separated.

Value Frameworks and Decision Frameworks

An important development in social decision making has been the development and use of “value frameworks” in the United States (Reference Anderson, Heidenreich and Barnett7;Reference Schnipper, Davidson and Wollins8;27). Our proposed framework differs from these in important ways. The most obvious is the concept of “value”. For most of these frameworks, the decision problem is characterized in fundamentally economic terms: value is determined by setting outcomes (variably defined) against cost (Reference Anderson, Heidenreich and Barnett7;Reference Schnipper, Davidson and Wollins8;27). Our approach fundamentally differs by considering value as one key attribute among four, rather than the central principle for decision making.

Portability of This Framework to Other Jurisdictions

We think there are aspects of our framework that will be useful internationally. We hope that thinking about HTA as an exercise in operationalizing social values will resonate elsewhere, and will lead to additional theoretical work that explicitly grounds technology decisions in social values. We also believe that representing appraisal criteria as factors that are embedded in social context and intellectual traditions is portable. We hope the simplicity, breadth, and practicability of this general approach will find it a wide audience.


This decision-making framework is being implemented in Ontario, Canada. Forthcoming publications will describe in greater detail the ongoing development and utility of this framework in real-world decision making.


Supplementary Table 1:


The authors have nothing to disclose.


This work was supported by the Ontario Ministry of Health and Long-Term Care and Health Quality Ontario. The views expressed in this publication do not necessarily represent the views of Health Quality Ontario and should not be construed as representing an official position.


1.Johnson, AP, Sikich, NJ, Evans, G, et al. Health technology assessment: A comprehensive framework for evidence-based recommendations in Ontario. Int J Technol Assess Health Care. 2009;25:141-50.CrossRefGoogle ScholarPubMed
2.Health Quality Ontario. Health technology assessments: Methods and process guide. Toronto, Ontario: Health Quality Ontario; 2017 p. 57Google Scholar
3.CADTH. pCODR Expert Review Committee: Deliberative Framework. 2016 March [16 pp.]. (accessed April 11, 2018).Google Scholar
4.Department of Health Clinical Access and Redesign Unit. The Queensland Policy and Advisory Committee on New Technology (QPACT): Decision making criteria. 2013:1. (accessed April 11, 2018).Google Scholar
5.Alonso-Coello, P, Oxman, AD, Moberg, J, et al. GRADE Evidence to Decision (EtD) frameworks: A systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. BMJ. 2016;353:i2089.CrossRefGoogle ScholarPubMed
6.Goetghebeur, MM, Wagner, M, Khoury, H, et al. Evidence and Value: Impact on DEcisionMaking: The EVIDEM framework and potential applications. BMC Health Serv Res. 2008;8:270.CrossRefGoogle ScholarPubMed
7.Anderson, JL, Heidenreich, PA, Barnett, PG, et al. ACC/AHA statement on cost/value methodology in clinical practice guidelines and performance measures: A report of the American College of Cardiology/American Heart Association Task Force on Performance Measures and Task Force on Practice Guidelines. J Am Coll Cardiol. 2014;63:2304-2322.CrossRefGoogle Scholar
8.Schnipper, LE, Davidson, NE, Wollins, DS, et al. American Society of Clinical Oncology Statement: A conceptual framework to assess the value of cancer treatment options. J Clin Oncol. 2015;33:2563-2577.CrossRefGoogle ScholarPubMed
9.Abelson, J, Wagner, F, DeJean, D, et al. Public and patient involvement in health technology assessment: A framework for action. Int J Technol Assess Health Care. 2016;32:256-264.CrossRefGoogle Scholar
10.National Institute for Health Care Excellence (NICE). NICE process and methods guides. Social value judgements: Principles for the development of NICE guidance. London: National Institute for Health and Care Excellence (NICE); 2008.Google Scholar
11.Stafinski, T, Menon, D, Marshall, D, et al. Societal values in the allocation of healthcare resources: Is it all about the health gain? Patient. 2011;4:207-225.CrossRefGoogle ScholarPubMed
12.Whitty, JA, Littlejohns, P. Social values and health priority setting in Australia: An analysis applied to the context of health technology assessment. Health Policy. 2015;119:127-136.CrossRefGoogle ScholarPubMed
13.Giacomini, M, Wagner, F, Krahn, M, et al. Social and ethical values for health technology assessment in Ontario. Health Quality Transformation 2012; Toronto, Ontario, 2012.Google Scholar
14.Miller, F. Improving the appraisal of non-drug technologies: Revising the Ontario Decision Framework Ethics & Social Values: Patient centred care. [Internet]. Saskatoon, SK: CADTH Symposium; 2015 [updated April 14, 2015; refd October 2, 2017]. (accessed April 11, 2018).Google Scholar
15.Miller, GA. The magical number seven plus or minus two: Some limits on our capacity for processing information. Psychol Rev. 1956;63:81-97.CrossRefGoogle ScholarPubMed
16.Redelmeier, DA, Rozin, P, Kahneman, D. Understanding patients' decisions. Cognitive and emotional perspectives. JAMA. 1993;270:72-76.CrossRefGoogle ScholarPubMed
17.Abelson, J, Forest, PG, Eyles, J, et al. Deliberations about deliberative methods: Issues in the design and evaluation of public participation processes. Soc Sci Med. 2003;57:239-251.CrossRefGoogle ScholarPubMed
18.Kuhn, TS. The structure of scientific revolutions. Chicago: University of Chicago Press; 1996.CrossRefGoogle Scholar
19.Dogan, M. Paradigms in the Social Sciences. In: Smelser, NJ, Baltes, PB, eds. International encyclopedia of the social & behavioral sciences. Oxford: Pergamon; 2001. p. 11023-11027.CrossRefGoogle Scholar
20.Devlin, N, Sussex, J. Incorporating multiple criteria in HTA: Methods and processes. London, UK: Office of Health Economics; 2011.Google Scholar
21.Claxton, K, Martin, S, Soares, M, et al. House of Commons, Parliamentary publication: Written Evidence (NICE 61). London, UK: Health Select Committee, 2012 October.Google Scholar
22.Keeney, RL. Utility functions for multiattributed consequences. Manage Sci. 1972;18(Pt-1):276-287.CrossRefGoogle Scholar
23.Schoemaker, PJH. The expected utility model: Its variants, purposes, evidence and limitations. J Econ Lit. 1982;20:529-563.Google Scholar
24.von Neumann, J, Morgenstern, O, Kuhn, HW, et al. Theory of games and economic behavior (60th Anniversary Commemorative Edition). Princeton, NJ: Princeton University Press; 1944.Google Scholar
25.National Institute for Health and Care Excellence (NICE). Guide to the processes of technology appraisal: Foreword. London, UK: The National Institute for Health and Care Excellence (NICE); 2014Google Scholar
26.Brouwer, WB, Culyer, AJ, van Exel, NJ, et al. Welfarism vs. extra-welfarism. J Health Econ. 2008;27:325-338.CrossRefGoogle ScholarPubMed
27.Institute for Clinical and Economic Review (ICER). Overview of the ICER value framework and proposals for an update for 2017–2018. Boston, MA: ICER; 2017. p. 1-24.Google Scholar
Figure 0

Figure 1. Social values relevant to the work of Ontario Health Technology Advisory Committee (OHTAC). EUnetHTA, European Network for Health Technology Assessment; HTA, health technology assessment; IJTAHC, International Journal of Technology Assessment in Health Care; INAHTA, International Nerwork of Agencies for Health Technology Assessment; MOHLTC, Ministry of Health and Long-Term Care.

Figure 1

Table 1. Contextual Factors and Health System Feasibility Components of the Revised Framework

Figure 2

Table 2. Benefits and Harms Component of the Revised Framework

Figure 3

Table 3. Economics and Patient-Centered Care Components of the Revised Framework

Supplementary material: File

Krahn et al. supplementary material

Table S1

Download Krahn et al. supplementary material(File)
File 30 KB
You have Access
Cited by

Send article to Kindle

To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Available formats

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Available formats

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *