Skip to main content Accessibility help
×
Home

Information:

  • Access
  • Cited by 4

Actions:

      • Send article to Kindle

        To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Pragmatic design in randomized controlled trials
        Available formats
        ×

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Pragmatic design in randomized controlled trials
        Available formats
        ×

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Pragmatic design in randomized controlled trials
        Available formats
        ×
Export citation

Abstract

At more than 10 years after the paper by Hotopf and colleagues regarding pragmatic trials in psychiatry, the field has evolved and is evolving further. There have been many developments in our understanding of what pragmatism really means, and excellent examples of truly pragmatic trials in psychiatry are currently available. Funders have helped encourage more emphasis on the need for such studies, but ‘local’ and trans-national regulations could help more. Consumers of the evidence should have a greater voice in generating the research agenda and, as this happens, the questions generated are more likely to be answered by a pragmatic approach to trials.

Introduction

It has been over 10 years since the paper of Hotopf et al. (1999) on pragmatic trials. This understandable call was partly a response to the increasing realization that even well-conducted randomized controlled trials did not necessarily address the needs of everyday care. Most trials involved participant groups so selected that they would be rare in routine care, and evaluated treatments administered so rigidly as to further remove the practice from reality and, finally, measured outcomes that were of questionable importance to clinicians, policymakers or recipients of care. The article by Hotopf et al. (1999) highlighted some of the benefits of pragmatic trials but with few examples to point to at that time. Most lessons had been learnt from colleagues in cancer, heart disease and perinatal medicine. Since that time, however, there have been many developments regarding our understanding of what pragmatism really means in trials and in how to conduct these types of studies in mental health. Further papers followed that highlighted a developing understanding of these types of studies within mental health care evaluation with increasing numbers of examples to cite (March et al. 2005; Stroup, 2011).

Despite this, ‘pragmatic’ remains a much misunderstood concept within trials and is often used as a broad, fashionable and, often, unhelpful label. Over the last years it became apparent that the concept of pragmatism in randomized trials was not binary (Stroup & Geddes, 2008). It has been recognized that considering randomized controlled trials as purely explanatory (studies that focused on whether, in ideal circumstances, a treatment would have any effect) or pragmatic (a study which investigated whether the treatment, given in real-world circumstances, really did have clinically meaningful effects) was too simplistic. The concept of explanatory and pragmatic trials being at opposite poles was dismissed in favour of the idea that there was a continuum between maximally pragmatic or fully explanatory.

Recently there has been a genuine advance in our understanding of pragmatism within trials (Thorpe et al. 2009). The authors defined 10 domains encapsulating explanatory and pragmatic approaches and covering the most important aspects of a trial design (Table 1).

Table 1. The 10 domains relevant to explanatory and pragmatic trials a

a Thorpe et al. (2009).

Thorpe et al. (2009) went on to define and explain each of these domains in detail and have left us with an elegant tool by which pragmatism could be estimated. These attractive diagrams (Fig. 1) can be created for those designing trials or assessing trials at the protocol stage of the funding application process to give some impression of applicability to clinical circumstances. Estimates are made regarding each parameter and this can result in ‘open’ or ‘closed’ diagrams illustrating how pragmatic or explanatory a trial is likely to be (Fig. 2).

Fig. 1. The blank Pragmatic–Explanatory Continuum Indicator Summary (PRECIS) ‘wheel’.

Fig. 2. Two examples of trials.

Understandably, this has triggered papers outlining specific tools for both trials and studies within systematic reviews (Tosh et al. 2011). These tools, with various degrees of reliability, can produce numerical estimates of the applicability of trials to the real world (Table 2).

Table 2. Scoring trial protocols estimating degrees of pragmatism a

ACHIEVE, Achieving healthy lifestyles in psychiatric rehabilitation; ERP, enhanced relapse prevention; DYD, Down Your Drink; PTSD – Yoga, Post-traumatic stress disorder – Yoga; CATIE, Clinical Antipsychotic Trials of Intervention Effectiveness; FIAT, Financial Incentives (for) Adherence Trial; TREC-SAVE, Tranquilizǎcção Rápida - Ensaio Clínic: Segurança no manejo da Agressão: Viabilidade e Ética na contenção e isolamento.

a From Tosh et al. (2011) and corrected to remove reference errors in the source table.

Mental health now has a series of trials to illustrate, to greater or lesser degrees, a more pragmatic approach. Previously the speciality had no choice but to employ explanatory trials in policy making. Currently, however, with increases in skills and experience, decision makers begin to have options. The last decade has seen a series of trials that falls much more into a pragmatic category. For example, Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE; Lieberman et al. 2005) and Cost Utility of the Latest Antipsychotic Drugs in Schizophrenia Study (CUtLASS; Jones et al. 2006) are important ground-breaking studies with greater levels of pragmatism than had previously been seen in such large mental health trials. Other randomized studies such as BALANCE (Bipolar Affective disorder: Lithium/ANticonvulsant Evaluation; Geddes et al. 2002) and the series of Tranquilizǎcção Rápida – Ensaio Clínic (TREC; Alexander et al. 2004; Huf et al. 2012) trials further developed techniques in pragmatism within mental health. All juggle the differing needs of researcher, clinician, policymaker and service user within a trial. In these studies the concept of pragmatism has been declined in different ways, but some key common characteristics include the following. First, all these studies investigated already marketed medicines, with the aim of answering comparative effectiveness. Second, the choice of control group interventions was mainly based on prevailing clinical practice. Third, patient populations were included minimizing exclusion criteria and being focused more on clinical presentation than on formal diagnoses. Fourth, the choice of outcome measures was based on pragmatic reasoning around what really outcome means for practising doctors. Finally, the sample size was calculated to show superiority, which implies that value is attributed to treatments that are associated with some degree of advantage over standard ones.

Clearly, critical issues may arise from such study designs. For example, the concept of ‘real life’ cannot be unambiguously defined, and therefore the results of pragmatic trials carried out in European countries might be hardly applicable to a very different system of care, such as, for example, a very low-income system of care. The same criticism may be applied to the choice of control treatments in pragmatic trials, as prevailing clinical practice may highly differ and therefore what may be relevant is a specific system of care may not be similarly relevant in others. Internal validity might be another critical aspect, as in some circumstances it is difficult to identify the ‘active ingredients’ of effective interventions assessed under a highly pragmatic design.

Progress has also been made in relation to reporting of trials. The CONSORT (Consolidated Standards of Reporting Trials) statement – a tool intended to improve quality of reporting and minimize risk of bias – has been extended to take into account pragmatism of trials (Zwarenstein et al. 2008). Additional text was added specifically for eight items, with detailed information on background, participants, interventions, outcomes, sample sizes, blinding, participant flow and generalizability of findings. These extensions represent an advance towards transparent and complete reporting of trials in general, and, in relation to the pragmatism of a trial, a guide for clinicians and policymakers for judging applicability to everyday clinical practice (Zwarenstein et al. 2008). This progress came from a change in culture, not least of which was in psychiatry, where farsighted researchers and clinicians anticipated the broad and hitherto often overlooked needs of the many stakeholders for trial data (Tansella et al. 2006).

The next 10 years

There is more to do. Legislation could dramatically change the landscape for pragmatic trials.

The regulatory authorities

The issue of pragmatic versus explanatory is a reflection of the aim or objectives of the trial. Pragmatic trials aim to provide answers to real-world questions perceived to be relevant to patients, clinicians and policymakers. Common to all such trials are measures of patient-valued outcomes. Regulatory trials do not necessarily aim to answer these types of questions. Regulatory trials aim to obtain regulatory approval. Currently, phase III trials are often carried out by industry and are highly explanatory, while phase IV trials may more often be carried out by groups of physicians who have a clinical dilemma and are more pragmatic (Barbui & Bighelli, 2013b ).

The Food and Drug Administration in the USA and the European Medicines Agency (EMA) are regulatory bodies that establish rules to get new medicines approved and marketed. The current explanatory standard of phase III studies, therefore, is a consequence of current rules and requirements issued by these agencies. Changing regulatory requirements could greatly encourage a more real-world and pragmatic set of designs within phase III studies (Wood, 2006). For example, in Europe new drugs can be evaluated with no comparison with active alternative treatments. As a consequence, phase III studies compare new drugs with placebo to make these new drugs eligible for registration – often not the question troubling the ‘real world’ of clinical care in which there may well be viable alternative drugs. If comparisons with other drugs are made within phase III studies, these usually rely on demonstrating therapeutic ‘non-inferiority’ – in agreement with current EMA requirements. If the scientific community agrees that active–control superiority is desirable, clinical trials should be designed and powered to generate the evidence and European pharmaceutical legislation should then be induced to incorporate this requirement, at least in addition to placebo-controlled trials (Garattini & Chalmers, 2009). It has recently been suggested that regulatory authorities may require at least one two-arm head-to-head pragmatic trial in order to demonstrate superiority of the investigational product over an active comparator (Barbui & Bighelli, 2013a ).

Conduct of studies

Increasingly, government funders of studies have become aware that the trials they have supported did not have the population-wide impact that they had intended (Duijnhoven et al. 2013). Explanatory studies are less likely to achieve wide impact for the general population because, by definition, those eligible for the explanatory study are a highly specific group of people from which most of the general population is excluded. As a result taxpayers’ money has funded studies that help far fewer taxpayers than had been originally intended. Broad-brush stipulation from funders – that real-world applicability is required in trial design (NHS National Institute for Health Research, 2013)  – and then authoritative governance to ensure that funding panels rigorously adhere to this stipulation can greatly help shift research culture and tradition. Additional legislation can help more. A case illustration helps. In Italy, the Gruppo Italiano per lo Studio della Streptochinasi nell'Infarto Miocardico (GISSI) trial was a highly pragmatic study investigating intravenous streptokinase in early acute myocardial infarction (Gruppo Italiano per lo Studio della Streptochinasi nell'Infarto Miocardico, 1986). This enrolled 11806 patients in 176 coronary care units. Subsequent to the first highly influential paper of 1986, debate focused around the need for more similar studies. In 2004 a Ministerial Decree was issued recognizing the public health value of the original study and the need of establishing rules to help implement pragmatic independent phase IV clinical trials (Tognoni & Franzosi, 2005). The Decree states that if a set of conditions is met (study coordinating centre is independent from drug companies, study results can be disseminated autonomously, there is no personal financial interest in studying the drugs included in the trial, study drugs are already in the market and are studied within the licensed indications), then the Italian health system supports the conduct of the trial in three ways. First, drug costs are paid by the Italian health system; (2) there are no fees for submitting the study protocol to the local ethics committees; (3) continuing medical education credits are provided to local investigators. The impact of this Decree on the development of pragmatic experimental studies has not been assessed yet, but from a cultural and theoretical viewpoint it has already been seen as a major advance in Italy.

Conclusions

When Hotopf et al. (1999) wrote about pragmatic trials in psychiatry their tone was almost wistful as regards other more advanced specialities. Things have changed and are changing further. We now have excellent examples of truly pragmatic trials in mental health. Funders have helped encourage more emphasis on the need for such studies but ‘local’ and trans-national regulations could help more. Consumers of the evidence should have a greater voice in generating the research agenda and, as this happens (Lloyd & White, 2011), the questions generated are, assuredly, more likely to be answered by a pragmatic approach to trials.

If we do not devalue the ‘pragmatic’ label through misuse (Sackett, 2013), the next decade will see a burgeoning of mental health trials with a much more emphasis on showing if treatments really work in everyday care.

Acknowledgements

This paper received no specific grant from any funding agency.

Declaration of Interest

None.

References

Alexander, J, Tharyan, P, Adams, C, John, T, Mol, C, Philip, J (2004). Rapid tranquillisation of violent or agitated patients in a psychiatric emergency setting. Pragmatic randomised trial of intramuscular lorazepam v. haloperidol plus promethazine. British Journal of Psychiatry 185, 6369.
Barbui, C, Bighelli, I (2013 a). A new approach to psychiatric drug approval in Europe. PLoS Medicine 10, e1001530.
Barbui, C, Bighelli, I (2013 b). Regulatory science in Europe: the case of schizophrenia trials. Lancet 382, 12341235.
Casagrande, SS, Jerome, GJ, Dalcin, AT, et al. (2010) Randomized trial of achieving healthy lifestyles in psychiatric rehabilitation: the ACHIEVE trial. BMC Psychiatry 10, 108.
Chalmers, I (1993). The Cochrane collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. Annals of the New York Academy of Sciences 703, 156–65.
Duijnhoven, RG, Straus, SM, Raine, JM, de Boer, A, Hoes, AW, De Bruin, ML (2013). Number of patients studied prior to approval of new medicines: a database analysis. PLoS Medicine 10, e1001407.
Garattini, S, Chalmers, I (2009). Patients and the public deserve big changes in evaluation of drugs. British Medical Journal 338, b1025.
Geddes, JR, Rendell, JM, Goodwin, GM (2002). BALANCE: a large simple trial of maintenance treatment for bipolar disorder. World Psychiatry 1, 4851.
Gold, C, Rolvsjord, R, Aaro, LE, Aarre, T, Tjemsland, L, Stige, B (2005). Resource oriented music therapy for psychiatric patients with low therapy motivation: protocol for a randomised controlled trial. BMC Psychiatry 5, 39.
Gruppo Italiano per lo Studio della Streptochinasi nell'Infarto Miocardico (GISSI) (1986). Effectiveness of intravenous thrombolytic treatment in acute myocardial infarction. Lancet i, 397402.
Hotopf, M, Churchill, R, Lewis, G (1999). Pragmatic randomised controlled trials in psychiatry. British Journal of Psychiatry 175, 217223.
Huf, G, Coutinho, ES, Adams, CE; TREC-SAVE Collaborative Group (2012). Physical restraints versus seclusion room for management of people with acute aggression or agitation due to psychotic illness (TREC-SAVE): a randomized trial. Psychological Medicine 42, 22652273.
Jones, PB, Barnes, TR, Davies, L, Dunn, G, Lloyd, H, Hayhurst, KP, Murray, RM, Markwick, A, Lewis, SW (2006). Randomized controlled trial of the effect on Quality of Life of second- vs first-generation antipsychotic drugs in schizophrenia: Cost Utility of the Latest Antipsychotic Drugs in Schizophrenia Study (CUtLASS 1). Archives of General Psychiatry 63, 10791087.
Lieberman, JA, Stroup, TS, McEvoy, JP, Swartz, MS, Rosenheck, RA, Perkins, DO, Keefe, RS, Davis, SM, Davis, CE, Lebowitz, BD, Severe, J, Hsiao, JK (2005). Effectiveness of antipsychotic drugs in patients with chronic schizophrenia. New England Journal of Medicine 353, 12091223.
Lloyd, K, White, J (2011). Democratizing clinical research. Nature 474, 277278.
Lobban, F, Gamble, C, Kinderman, P, Taylor, L, Chandler, C, Tyler, E, Peters, S, Pontin, E, Sellwood, W, Morriss, RK (2007). Enhanced relapse prevention for bipolar disorder – ERP trial. A cluster randomised controlled trial to assess the feasibility of training care coordinators to offer enhanced relapse prevention for bipolar disorder. BMC Psychiatry 7, 6.
March, JS, Silva, SG, Compton, S, Shapiro, M, Califf, R, Krishnan, R (2005). The case for practical clinical trials in psychiatry. American Journal of Psychiatry 162, 836846.
Morriss, R, Marttunnen, S, Garland, A, Nixon, N, McDonald, R, Sweeney, T, Flambert, H, Fox, R, Kaylor-Hughes, C, James, M, Yang, M (2010). Randomised controlled trial of the clinical and cost effectiveness of a specialist team for managing refractory unipolar depressive disorder. BMC Psychiatry 10, 100.
Murray, E, McCambridge, J, Khadjesari, Z, White, IR, Thompson, SG, Godfrey, C, Linke, S, Wallace, P (2007). The DYD-RCT protocol: an on-line randomised controlled trial of an interactive computer-based intervention compared with a standard information website to reduce alcohol consumption among hazardous drinkers. BMC Public Health 7, 306.
NHS National Institute for Health Research (2013). HTA programme remit. The Health Technology Assessment Programme (http://www.hta.ac.uk/funding/remit.shtml). Accessed 30 October 2013.
Priebe, S, Burton, A, Ashby, D, Ashcroft, R, Burns, T, David, A, Eldridge, S, Firn, M, Knapp, M, McCabe, R (2009). Financial incentives to improve adherence to anti-psychotic maintenance medication in non-adherent patients – a cluster randomised controlled trial (FIAT). BMC Psychiatry 9, 61.
Sackett, DL (2013). Clinician-trialist rounds: 17. Mind your explanatory and pragmatic attitudes! Part 2: How? Clinical Trials 10, 633636.
Stroup, TS (2011). What can large simple trials do for psychiatry? American Journal of Psychiatry 168, 117119.
Stroup, TS, Geddes, JR (2008). Randomized controlled trials for schizophrenia: study designs targeted to distinct goals. Schizophrenia Bulletin 34, 266274.
Stroup, TS, McEvoy, JP, Swartz, MS, Byerly, MJ, Glick, ID, Canive, JM, McGee, MF, Simpson, GM, Stevens, MC, Lieberman, JA (2003). The National Institute of Mental Health Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) project: schizophrenia trial design and protocol development. Schizophrenia Bulletin 29, 1531.
Tansella, M, Thornicroft, G, Barbui, C, Cipriani, A, Saraceno, B (2006). Seven criteria for improving effectiveness trials in psychiatry. Psychological Medicine 36, 711720.
Telles, S, Singh, N, Joshi, M, Balkrishna, A (2010). Post traumatic stress symptoms and heart rate variability in Bihar flood survivors following yoga: a randomized controlled study. BMC Psychiatry 10, 18.
Thorpe, KE, Zwarenstein, M, Oxman, AD, Treweek, S, Furberg, CD, Altman, DG, Tunis, S, Bergel, E, Harvey, I, Magid, DJ, Chalkidou, K (2009). A Pragmatic–Explanatory Continuum Indicator Summary (PRECIS): a tool to help trial designers. Canadian Medical Association Journal 180, E47E57.
Tognoni, G, Franzosi, MG (2005). [Experimenting in Italy: from GISSI to the Ministerial Decree]. Recenti Progressi in Medicina 96, 9294.
Tosh, G, Soares-Weiser, K, Adams, CE (2011). Pragmatic vs explanatory trials: the Pragmascope tool to help measure differences in protocols of mental health randomized controlled trials. Dialogues in Clinical Neuroscience 13, 209215.
Wood, AJ (2006). A proposal for radical changes in the drug-approval process. New England Journal of Medicine 355, 618623.
Zwarenstein, M, Treweek, S, Gagnier, JJ, Altman, DG, Tunis, S, Haynes, B, Oxman, AD, Moher, D (2008). Improving the reporting of pragmatic trials: an extension of the CONSORT statement. British Medical Journal 337, a2390.