Introduction
The Scottish Medicines Consortium (SMC) undertakes health technology assessment (HTA) to decide whether newly licensed medicines can be accepted for routine use by NHSScotland. There is a two-stage assessment process for each medicine: first, a New Drugs Committee (NDC) considers the pharmaceutical company’s submission of the clinical and health economic evidence; in the second stage, the SMC committee considers the NDC’s preliminary advice together with a broader range of evidence to decide whether the medicine is an acceptable use of limited NHS resources. Committee decisions are based on a majority anonymous vote.
Following concern about low acceptance rates for medicines used to treat rare conditions and those used for conditions with short life expectancy, SMC introduced the Patient and Clinician Engagement (PACE) process in 2014 (Reference Morrell, Wordsworth, Fu, Rees and Barker1). PACE is an additional stage in the assessment process for end-of-life (EoL), orphan, and ultra-orphanFootnote 1 medicines (2). EoL medicines are defined by SMC as those used to treat conditions at a stage that usually leads to death within 3 years with currently available treatments; orphan medicines are used to treat conditions affecting fewer than 5 people in a population of 10,000; and, ultra-orphan medicines are used for conditions with a prevalence of 1 in 50,000 or less. If the NDC’s preliminary advice is not to recommend the medicine, the submitting company may request a PACE meeting. The aim of PACE is to describe the added benefits of the medicine that may not be fully captured within the conventional HTA process. The PACE meeting provides an opportunity for patient groups (PG) and clinicians with a specialist knowledge of a condition to agree on a consensus statement about the added value of the new medicine from their perspective. PG and clinicians submit written statements in advance of the PACE meeting to support discussion. The consensus PACE statement is provided in SMC committee members’ papers and a verbal summary of the key points is presented at the SMC committee meeting.
The addition of PACE, alongside other process changes introduced by SMC at the same time, has resulted in a higher acceptance rate for orphan and cancer medicines (3).
Despite the recognition that patients, carers, and clinicians can play a valuable role in strengthening HTA (Reference Angelis and Kanavos4;Reference Gauvin, Abelson, Giacomini, Eyles and Lavis5), few studies have investigated how HTA decision makers use this information (Reference Weeks, Polisena and Scott6). There is evidence that patient insights can help committee members interpret HTA evidence (Reference Berglas, Jutai, Mackean and Weeks7), and directly influence the content of HTA reports (Reference Dipankui, Gagnon and Desmartis8), although the extent of this influence is unclear. Previous studies have also shown great variation in how committee members approach “experiential evidence” of patient testimonies (Reference Hashem, Calnan and Brown9). To understand how the evidence gathered through PACE supports decision making, SMC conducted an evaluation of the PACE process. The first stage of the evaluation was conducted in 2016 and involved a thematic analysis of 28 PACE statements (supporting the assessment of EoL and orphan medicines reviewed by SMC between September 2014 and August 2015) to gain a better understanding of the information that is captured through the PACE process. Three prominent themes were identified (in order of prominence): impact on quality of life (QoL); impact on life expectancy; and addresses an unmet need (Reference Jones, Harkess-Murphy and Lockhart10).
Eight subthemes were identified for the most prominent theme, impact on QoL:
-
• Ability to take part in normal life
-
• Convenience of administration
-
• Health benefits
-
• Impact on families and carers
-
• Opportunity for treatment choice
-
• Providing hope for the future
-
• Psychological benefits
-
• Tolerability
The second phase of the evaluation explored the importance of these factors (i.e., the three prominent themes and QoL subthemes) to SMC committee members and PACE participants.
Methods
Two online questionnaires were developed to collect information on the perceived importance of factors (i.e., themes and QoL subthemes) in PACE statements: one for SMC committee members (Supplementary File 1) and one for PACE participants (Supplementary File 2).
The first questionnaire asked SMC committee members to rate the importance of each factor when making a decision on an orphan, ultra-orphan, and/or EoL medicine (in general). The second questionnaire asked PACE participants how important they believe the factors should be to SMC committee members when making a decision. Respondents were also asked whether they would rate the importance of the themes/subthemes differently depending on whether the medicine had orphan, ultra-orphan, and/or EoL status.
The level of importance was captured using a four-point ordered categorical rating scale (very important, important, of lesser importance, and not important).
The questionnaires also collected information about respondents’ experience with SMC and PACE, to explore whether those with more exposure to the PACE process (for example, long-serving committee members or those who have attended multiple PACE meetings) had different views to those with less exposure (for example, new committee members or those who had attended only one PACE meeting).
The first survey was conducted in March 2019. At the time of the survey, the SMC committee consisted of 26 members who were invited to complete the online questionnaire. The second survey was conducted in January 2020. All PACE participants (42 PG representatives and 56 clinicians) who participated in one or more PACE meetings during 2019 were invited to complete the online questionnaire. A total of 31 PACE meetings were held in 2019.
Data analysis
Analysis of the ordered categorical survey data used mainly descriptive techniques. First, the percentage of respondents who rated the themes at each level of importance (i.e., very important, important, of lesser importance, or not important) was calculated and diverging stack charts were generated. The themes were then ranked in order of importance. The same process was used to analyze the importance of the QoL subthemes. The percentage of respondents who would rate the themes/QoL subthemes differently depending on a medicine’s status was then calculated. Content analysis was used to analyze free-text responses.
Results
SMC committee members
Of the 26 SMC committee members invited to participate, 21 members responded. One response was excluded from the analysis because the survey was only partially completed, giving a total response rate of 77 percent (n = 20).
The importance of factors in PACE statements to SMC committee members
Overall, the three themes in PACE statements (QoL, life expectancy, and unmet need) were considered very important or important by the majority of committee members. The theme with the highest rating was impact on QoL, with 55 percent of respondents (n = 11) indicating that it was very important and 45 percent (n = 9) indicating that it was important. Impact on life expectancy was also considered important, with only one respondent rating this theme of lesser importance. However, addresses an unmet need received a more mixed response, with 40 percent (n = 8) rating this as very important, 35 percent (n = 7) as important and 25 percent (n = 5) as of lesser importance.
Quality of life subthemes
Committee members rated the importance of six of the eight QoL subthemes more positively (predominantly very important or important) and two more negatively (predominantly of lesser importance or not important). Health benefits received the highest rating and opportunity for treatment choice received the lowest rating (Figure 1).
Figure 1. Importance of quality of life (QoL) subthemes to SMC committee members.
Consensus among committee members was highest for health benefits with 80 percent (n = 16) of respondents indicating that it is very important. The most varied responses were received for providing hope for the future, with 18 percent (n = 3), 35 percent (n = 6), 41 percent (n = 7), and 6 percent (n = 1) rating this QoL subtheme as very important, important, of lesser importance, and not important, respectively.
No respondent gave the same rating to all eight QoL subthemes.
Length of time served on SMC committee
Of those who responded, four committee members had served on the SMC committee for less than 1 year, nine members for 1 to 3 years, and seven members for longer than 3 years. Due to small numbers, any association between time served on the SMC committee and responses to the questions would be difficult to determine and are not presented.
Medicine status
Approximately 60 percent of respondents (n = 12) reported that they would not rate the importance of themes differently depending on a medicine’s status, that is, orphan, ultra-orphan, or EoL medicine. Respondents explained that the focus on the patient and the health impact would remain the same, regardless of the medicine’s status. One respondent commented: As it is the individual patient at the heart of each decision I do not feel that there should be any distinction based on the categorization of the medicine in what realistically is being used for an incurable condition.
Another respondent commented: I feel regardless of the disease area or nature of the disease, the health impact (captured in quality of life and life years) of technologies is the most important consideration in the context of constrained health service resources. For truly rare diseases with currently no available treatments, it is probably reasonable to accept a higher cost-effectiveness threshold – but demonstrable health benefits remain the most important consideration—rather than addressing the unmet need for the sake of it.
Twenty percent of respondents (n = 4) indicated that they would maybe rate the themes differently under certain circumstances. One respondent explained that it would depend on how significantly the medicine was anticipated to impact each theme. A further four respondents indicated that they would rate the themes differently depending on the medicine’s status. Respondents mainly described differences between orphan and EoL medicines, with one explaining that although QoL is equally important regardless of status, the importance of unmet need and life expectancy may change depending on whether the medicine is an orphan or EoL medicine. One respondent said: The life expectancy criterion may not apply to orphan and ultra-orphan, whereas QoL always applies. Another respondent stated: For end-of-life, unmet need would be of lesser importance. Impact on quality of life would be very important.
As with the themes in PACE statements, 60 percent of respondents (n = 12) reported that they would not rate the QoL subthemes differently based on a medicine’s status. Thirty-five percent of respondents (n = 7) indicated that they would maybe rate the QoL subthemes differently. One explanation provided was: Many of these things are quite interlinked. In the case of end of life, it is perhaps more important to be assured that any limited extension to life is lived with reasonable quality of life.
Only one respondent reported that they would rate the QoL subthemes differently. They explained: Profile would be very different in the case of end of life.
Other themes or elements in PACE statements of importance to SMC committee members
Respondents were asked whether there were any other themes or elements of PACE statements that were important in their decision making. Of five respondents, two stated that they value information about the level of support for the medicine from patients and clinicians. As one respondent explained: Information from clinical experts is very helpful, particularly if negative or there is lack of enthusiasm for the drug being assessed.
Respondents also highlighted that receiving information about the experience of living with the condition complements QoL scores: Description of the impact of the condition on a patient’s day-to-day life is very helpful as it fleshes out the data from the quality-of-life scores.
Information on whether the medicine can be delivered to sites close to patients was also useful to one respondent. Finally, describing the value of PACE statements more generally, one respondent said: I find the PACE statements a key element of the process and I am often influenced by the impact of medications on people’s real-life experience of managing their illness. It is never the sole influence but it is significant.
PACE participants
A total of 40 PACE participants responded, comprising 26 PG representatives and 14 clinicians (response rates of 62 percent and 25 percent, respectively). Of the 26 PG respondents, 23 percent (n = 6) indicated that they were patients with lived experience of the condition.
The importance of factors in PACE statements to PACE participants
The majority of PACE participants felt that the factors in PACE statements should be important or very important to committee members when they are making a decision about a PACE medicine. Impact on QoL received the highest rating with 88 percent (n = 35) indicating that it is very important versus 75 percent (n = 30) for impact on life expectancy and 78 percent (n = 31) for addresses an unmet need.
Quality of life subthemes
The majority of PACE participants rated the importance of the QoL subthemes as either important or very important (Figure 2). Similar to committee members, the themes that received the highest ratings were ability to take part in normal life and health benefits and the two QoL subthemes to receive the lowest ratings were opportunity for treatment choice and convenience of administration.
Figure 2. Importance of quality of life (QoL) subthemes to Patient and Clinician Engagement (PACE) participants.
The highest level of agreement among PACE participants was for the importance of ability to take part in normal life, with 88 percent (n = 35) rating this subtheme as very important. Providing hope for the future generated the most mixed response among PACE participants, with 42 percent (n = 17), 40 percent (n = 16), and 18 percent (n = 7) indicating that it was very important, important, and of lesser importance, respectively.
Experience with the PACE process
The majority of PACE participants (n = 23; 58 percent) had attended only one PACE meeting, 13 percent (n = 5) had attended two meetings, and 30 percent (n = 12) had attended three or more meetings.
Most PACE meetings attended by PG representatives and clinicians, who responded to the questionnaire, were for orphan medicines (18 and 9 meetings, respectively) compared with ultra-orphan (2 and 1 meetings, respectively) or EoL medicines (8 and 5 meetings, respectively). A total of three PG representatives and one clinician did not know the status of the medicine (i.e., orphan, ultra-orphan, or EoL medicine) discussed at the meeting. In general, the overall pattern of responses was similar between the two groups.
Medicine status
A small majority of respondents (n = 17; 43 percent) reported that they would not rate the importance of themes differently depending on a medicine’s status (i.e., orphan, ultra-orphan, or EoL medicine). Several free-text explanations given to support this response described how QoL factors are important regardless of the medicine’s status. A clinician explained: From a patient’s perspective the impact that a medicine would have on their quality of life to a large part is independent of whether this is an orphan, ultra-orphan, or end-of-life medicine.
Twenty-five percent of respondents (n = 10) indicated that they would maybe rate the importance of the themes differently depending on the medicine status. Three respondents explained that the importance of QoL information is more important for EoL medicines than orphan or ultra-orphan medicines, as one clinician explained:
Quality of life is important in all settings but paramount in the end-of-life setting. A PG representative said: Maintaining a good quality of life is absolutely crucial regardless of whether or not a condition is common or extremely rare. It could be argued that it is even more important for individuals receiving end-of-life medicine/care as they will want the opportunity to get their affairs in order and make the most of the time they have left.
Similarly, respondents who indicated that they would rate the QoL themes differently depending on the medicine’s status generally explained that certain QoL subthemes are more important for EoL medicines, for example, treatment choice and psychological benefits. One PG representative explained: Quality of life and psychological benefits would be more important for end-of-life medicine.
Other information in PACE statements that PACE participants consider should be important in SMC decision making
PACE participants were asked if any other QoL information should be important in SMC decision making. Two respondents highlighted that each patient is different with different needs with one further respondent suggesting that patient-reported experience should be considered. A further respondent noted that: In cancer, QoL will fall with the best supportive care as the disease progresses, and therefore weighing this up with a progression free or overall survival benefit with a drug that may reduce QoL is key. More generally, a respondent commented that: It is important that the patients’ voice is not just a box-ticking exercise in the form of PACE.
Difference in responses between PACE participants (PG representatives versus clinicians)
Overall clinicians and PG representatives ranked the importance of the themes and QoL subthemes similarly. For both groups, the highest-rated QoL subthemes were health benefits and the ability to take part in normal life and the lowest-rated were convenience of administration and opportunity for treatment choice (Figure 3). However, PG representatives generally rated the subthemes higher than clinicians, with the exception of tolerability, which was rated very important by 50 percent of clinicians (n = 7) and only 35 percent of PG representatives (n = 9). The greatest difference of opinion was about the importance of providing hope for the future, which 58 percent of PG representatives rated as very important (n = 15) compared with only two clinicians. There did not appear to be any differences between PG representatives who were patients with experience of the condition (n = 6) and those who were representing a PG (n = 20).
Figure 3. Importance of quality of life (QoL) subthemes to patient group (PG) representatives versus clinicians.
Comparison of responses from PACE participants and SMC committee members
In general, PACE participants rated the importance of themes (Figure 4) and QoL subthemes (Figure 5) higher than SMC committee members. However, the rank order of themes, from highest rated to lowest rated, is very similar for PACE participants and SMC committee members. This might indicate that the relative importance of themes and subthemes is similar for both groups.
Figure 4. Importance of Patient and Clinician Engagement (PACE) themes for SMC committee members versus PACE participants.
Figure 5. Importance of quality of life (QoL) subthemes: SMC committee members versus Patient and Clinician Engagement (PACE) participants.
Health benefits had the highest level of agreement between committee members and PACE participants. The greatest differences between committee members and PACE participants were for treatment choice and providing hope for the future (Figure 5).
The majority of committee members and PACE participants (60 percent versus 45 percent) indicated that they would not rate the importance of the QoL subthemes differently depending on whether the medicine was an orphan, ultra-orphan, and/or EoL. However, compared with committee members, a higher proportion of PACE participants (30 percent versus 5 percent) indicated that they would rate the QoL subthemes differently. Both committee members and PACE participants who indicated they would maybe rate the subthemes differently (10 percent versus 50 percent) supported this response by highlighting the different needs of patients with EoL conditions compared with rare/very rare conditions.
Discussion
This study demonstrated that, in general, PACE themes and subthemes that were rated highly by PACE participants were also considered to be important by SMC committee members. The findings indicate that information provided by PACE statements is of value to SMC committee members when they are making decisions on new orphan, ultra-orphan, and/or EoL medicines.
Committee member responses were largely heterogeneous and no two respondents provided exactly the same answers to all questions. This suggests that SMC committee members use information in PACE statements in different ways. For SMC committee members, the information in PACE statements can help to provide context, making health-related QoL scores collected in clinical studies more meaningful, as highlighted by some free text responses. Furthermore, certain factors captured in PACE statements (such as impact on life expectancy, unmet need, health benefits, and tolerability) are often included as part of other evidence considered by SMC. However, by presenting this information in a different format (such as in PACE statements and verbal PACE presentations) PACE could help some committee members make better use of this information.
Overall, committee members considered the importance of the PACE themes and QoL subthemes less positively than PACE participants. However, it is possible to observe some alignment between committee members’ responses and PACE participants’ responses, from subthemes that received the highest ratings of importance to those that received the lowest ratings. This might indicate that, to some extent, decision making on PACE medicines reflect the priorities of PACE participants. Regardless, when SMC committee members make a decision on any new medicine, they have to consider a broad range of evidence and take into account the needs of all patients in Scotland, not just those with the condition under review. Therefore it is not surprising that the importance ratings were generally lower for committee members. In addition, no respondent gave the same rating to all eight QoL subthemes suggesting that each respondent differentiated between the levels of importance in the scale.
An interesting finding to emerge from the analysis of PACE participant responses was the difference between PG representative and clinician ratings of importance. This finding supports the value of having both groups represented at the PACE meeting in order to capture a broad range of insights.
The importance of providing hope for the future generated particularly divergent responses between PG representatives and clinicians, as well as committee members. PG representatives rated this theme more highly. This may reflect differences in how hope is perceived by patients and clinicians. For example, palliative patients may have strong hope despite knowing their prognosis. In contrast, health professionals are likely to have a more realistic perspective on hope, informed by the clinical evidence base (Reference Nierop-van Baalen, Grypdonck, Van Hecke and Verhaeghe11;Reference Nierop-van Baalen, Grypdonck, Van Hecke and Verhaeghe12).
Interestingly, three committee members chose not to rate the importance of providing hope for the future. One explanation could be that the theme was not well understood, perhaps because of its close associations with psychological benefit. Alternatively, they might not have had a strong opinion but there was no neutral answer option. It could also reflect social desirability bias, if respondents had a negative view which they did not want to express. To ensure that providing hope for the future is described in an impactful way, PACE chairs and SMC staff involved in producing PACE statements should consider how best to discuss hope at PACE meetings and incorporate this theme into PACE statements.
There were some limitations in this study. Although the response rate across the two surveys was high, it was notably lower for clinician PACE participants compared to PG representatives and SMC committee members. Time and workload pressures in clinical practice may account for the lower response; motivation to respond could also be less in this group (Reference Booker, Austin and Balasubramanian13).
The original thematic analysis, from which the PACE themes were extracted, was conducted in 2016 using PACE statements from 2014 and 2015. It is possible that 2019 themes may differ from 2014/15 themes due to, for example, more experience with the PACE process, views of different PACE participants, and changes to SMC processes. For example, a new pathway for ultra-orphan medicines was introduced in April 2019 whereby PACE meetings are no longer conducted at the time of initial assessment.
The majority of PACE participants who responded to the questionnaire had little experience of the PACE process and may have been unfamiliar with HTA decision making for new medicines. The questionnaire did not allow respondents to indicate if they were unsure or undecided on the importance of individual factors to decision making or include a neutral answer option. Although this could be viewed as a limitation, these options were excluded to encourage opinions. No participant raised an objection to the 4-point rating scale.
In addition, to preserve anonymity and encourage candid responses the length of time served on the SMC committee was the only demographic information collected from committee members. Although this strategy limited the analysis, it ensured a high response rate from committee members. However, the sample sizes were relatively small, limiting the use of statistical tests to explore patterns of responses and interpretation of the findings.
Finally, this study has reported on subthemes that received the highest and lowest ratings of importance but it is important to note that subthemes in PACE statements are closely linked. As Morrell et al. (Reference Morrell, Wordsworth, Fu, Rees and Barker1) observed, the themes to emerge from PACE are interrelated and should not be viewed in isolation, for example, opportunity for treatment choice may provide hope for the future, which may in turn reduce anxiety and improve psychological outcomes.
Conclusions
This study has provided the first insights into patient-based information that committee members consider to be important in their decision making and demonstrates the value of PACE and the importance of including different stakeholders in the PACE process. The findings suggested that despite differences in how PACE participants and SMC committee members rated the importance of the themes, there was some alignment between the responses, supporting the value of the PACE output in decision making.
To gain a better understanding of how committee members use evidence to make decisions, future research might explore individual preferences for types of evidence and how it is presented. By understanding what information is most valuable to committee members and PACE participants, these findings will help SMC to optimize use of the PACE process.
Supplementary materials
To view supplementary material for this article, please visit http://doi.org/10.1017/S026646232300003X.
Acknowledgments
We would like to extend our thanks to the SMC committee members and PACE participants who responded to the questionnaires for this study.
Funding statement
This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
Conflicts of interest
The authors declare none.
