Skip to main content Accessibility help
×
×
Home

Information:

  • Access
  • Open access

Actions:

      • Send article to Kindle

        To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Hiding negative trials by pooling them: a secondary analysis of pooled-trials publication bias in FDA-registered antidepressant trials
        Available formats
        ×

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Hiding negative trials by pooling them: a secondary analysis of pooled-trials publication bias in FDA-registered antidepressant trials
        Available formats
        ×

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Hiding negative trials by pooling them: a secondary analysis of pooled-trials publication bias in FDA-registered antidepressant trials
        Available formats
        ×
Export citation

Abstract

Background

Previous studies on reporting bias generally examined whether trials were published in stand-alone publications. In this study, we investigated whether pooled-trials publications constitute a specific form of reporting bias. We assessed whether negative trials were more likely to be exclusively published in pooled-trials publications than positive trials and examined the research questions, individual trial results, and conclusions presented in these articles.

Methods

Data from a cohort of 105 randomized controlled trials of 16 antidepressants were extracted from earlier publications and the corresponding Food and Drug Administration (FDA) reviews. A systematic literature search was conducted to identify pooled-trials publications.

Results

We found 107 pooled-trials publications that reported results of 23 (72%) of 32 trials not published in stand-alone publications. Only two (3.8%) of 54 positive trials were published exclusively in pooled-trials publications, compared with 21 (41.1%) of 51 negative trials (p < 0.001). Thirteen (12%) of 107 publications had as primary aim to present data on the trial's primary research question (drug efficacy compared with placebo). Only four of these publications, reporting on five (22%) trials, presented individual efficacy data for the primary research question. Additionally, only five (5%) of 107 pooled-trials publications had a negative conclusion.

Conclusions

Compared with positive trials, negative trials of antidepressants for depression were much more likely to be reported exclusively in pooled-trials publications. Pooled-trials publications flood the evidence base with often-redundant articles that, instead of addressing the original primary research question, present (positive) results on secondary questions. Therefore, pooled-trials publications distort the apparent risk–benefit profile of antidepressants.

Introduction

Reporting bias has been demonstrated in many medical fields (McGauran et al., 2010; Hart et al., 2012; Dwan et al., 2013). An important form of reporting bias is study publication bias, which occurs when trials with positive results are more likely to be published than those with negative results (Higgins and Green, 2011). In studies on reporting bias, trials that are published exclusively in pooled-trials publications, which pool data from multiple trials, are usually regarded as unpublished (Turner et al., 2008, 2012) or incompletely published (Rising et al., 2008). However, some pharmaceutical companies have argued that these trials have actually been published (Lilly press release 2008).

Although pooled-trials publications can provide new information, they may be particularly susceptible to bias, for example, because it is often unclear how trials were selected for inclusion (Thaler et al., 2013). The research question of pooled-trials publications also often differs from the original research question. For example, they may focus on differential efficacy in patient subgroups, leading to substantial redundancy and the suggestion that many of these articles represent ‘salami publications’ (Spielmans et al., 2010, 2017). Additionally, negative trials, in contrast to positive trials, may be published exclusively in pooled-trials publications. A study examining trials for five antidepressants approved between 1989 and 1994 found that positive trials were usually reported in stand-alone publications, while negative trials were frequently ‘bundled’ into pooled-trials publications (Melander et al., 2003). Consequently, pooled-trials publications may actually further bias the published literature and the apparent risk–benefit profile of treatments, rather than helping to provide transparent access to trial results. However, that study (Melander et al., 2003) did not assess the research questions of pooled-trials publications, whether they were likely to reach positive conclusions, and whether individual trial results were also presented.

A previous meta-analysis found that 31% of antidepressant trials for major depressive disorder remained unpublished (Turner et al., 2008). However, pooled-trials publications were excluded from that study. In the present study, we use that meta-analysis and data from four novel antidepressants approved subsequently to investigate whether the practice of pooling trials constitutes a specific form of reporting bias. We assessed whether unpublished trials were actually published in pooled-trials publications and determined how frequently negative trials were published exclusively in pooled-trials publications compared with positive trials. Secondly, we evaluated whether the research question of pooled-trials publications corresponded to the original primary research question of the included trials and whether these publications reported individual trial results for this primary outcome. Finally, we assessed how often pooled-trials publications reached positive conclusions.

Methods

FDA-registered trials

Information on phase 2/3 clinical trial programs for 16 second-generation antidepressants (bupropion sustained release, citalopram, desvenlafaxine, duloxetine, escitalopram, fluoxetine, levomilnacipran, mirtazapine, nefazodone, paroxetine immediate release, paroxetine controlled release, sertraline, venlafaxine immediate release, venlafaxine extended release, vilazodone, and vortioxetine) was extracted from an earlier publication (Turner et al., 2008) or from Food and Drug Administration (FDA) reviews using the same methodology. These reviews can also be accessed online (Turner, 2013a; OHSU Digital Commons, n.d.). Because pharmaceutical companies must preregister trials they intend to conduct in support of US marketing approval, FDA reviews can be used as a registry and results database (Turner, 2004).

These programs included 105 trials investigating the short-term treatment of depression. Consistent with Turner et al. (2008), we extracted the FDA's regulatory decision [i.e. whether the primary endpoint(s) were judged to be positive]; 54 trials were considered positive and 51 trials not-positive in the current study. Detailed information on these trials is provided by Turner et al. (2008). We included only FDA-registered trials because this enabled us to assemble a complete cohort of premarketing trials; although some pharmaceutical companies have detailed trial registries (e.g. GlaxoSmithKline), it is generally difficult or impossible to obtain information on older, unpublished trials that are not FDA-registered.

Trials published as stand-alone publications

We retrieved the references of 50 journal articles reporting the results of FDA-registered antidepressant trials from Turner et al. (2008). One article presented the pooled results of two identically designed trials of paroxetine controlled release (Golden et al., 2002); we regarded this article as a pooled-trials publication and the included trials were considered unpublished in stand-alone form. Additionally, we found a matching stand-alone publication (Miller et al., 1989) for one trial of paroxetine (UK-06) considered unpublished by Turner et al., and we found 23 stand-alone articles reporting the results of novel antidepressant trials.

Trials published in pooled-trials publications only

We assessed whether trials not published in stand-alone form were published in pooled-trials publications. Pooled-trials publications were defined as publications in which the individual patient data of two or more trials were analyzed. This included publications described as individual patient data meta-analyses, but it did not include meta-analyses based on aggregate data. A systematic literature search was conducted in PubMed, EMBASE, and the Cochrane Central Register of Controlled Trials, restricted to articles in English, until 15 August 2017. The search strategy included the drug, depression-related terms, and ‘placebo’. Terms were customized to each database; for example, the search string for searching PubMed for citalopram publications was: citalopram [Title] AND depress* [Title/abstract] AND placebo. After identifying pooled-trials publications, trial matches were identified using the drug name, active comparator (when applicable), dosage groups, sample sizes, trial duration, and names of investigators. We only included pooled-trials publications for which included trials could be matched to FDA-registered trials. From each publication, we extracted the primary endpoint, which was categorized as ‘primary efficacy’, i.e. the pooled-trials publication endpoint was the same as the original trial's primary endpoint (efficacy of the drug compared with placebo), or ‘not primary efficacy’. The second category consisted of secondary efficacy outcomes (e.g. anxiety, efficacy compared with an active comparator), predictors of efficacy (e.g. efficacy in subgroups, baseline severity), and other efficacy or safety outcomes. We also extracted whether publications reported individual trial results for the original primary outcome of the included trials. Additionally, AR and YV classified each pooled-trials publication as positive, neutral, or negative, based on the abstract (96% inter-rater agreement). Differences were resolved by consensus. Publications were considered positive when the abstract claimed that the antidepressant was more effective than placebo or an active comparator; equally effective as an active comparator; safer, better tolerated, or equal in safety/tolerability to placebo or an active comparator; or simply ‘safe’ or ‘well-tolerated’. Publications were considered neutral when the publication was primarily methodological in orientation or otherwise did not address the antidepressant in question.

Statistical analysis

We examined whether not-positive trials were more likely to be published exclusively in pooled-trials publications than positive trials. Because of small cell sizes, p values were obtained with Fisher's exact test, using Stata (version 13.1).

Results

Pooled-trials publications

As shown in the flow diagram (Fig. 1), of 105 FDA-registered antidepressant trials, 32 were not published in stand-alone publications. Of these, 23 (71.9%) were included in 107 pooled-trials publications (Table 1).

Fig. 1. Flowchart of trial inclusion and pooled-trials publication inclusion.

Table 1. Number of pooled-trials publications and their research questions for all trials not published in stand-alone form

CR, controlled release; FDA, Food and Drug Administration; IR, immediate release; XR, extended release.

Of the 51 not-positive trials, 21 (41.1%) were exclusively published in pooled-trials publications, compared with two (3.7%) positive trials (Fig. 2). Consequently, all positive trials were published in some form (either stand-alone or pooled), as were 82.4% of not-positive trials. Compared with positive trials, not-positive trials were more likely to be published exclusively in pooled-trials publications (Fisher's exact p < 0.001).

Fig. 2. Publication status of positive and not-positive FDA trials.

Characteristics of pooled-trials publications

Thirteen of 107 (12.1%) pooled-trials publications, including 10 (43%) of 23 trials published exclusively in pooled-trials publications, had the same research question as the included trial's original primary research question (drug efficacy compared with placebo) (Table 1). Only four (3.7%) publications presented individual efficacy data for the primary research question, reporting efficacy results for five (21.7%) trials. Other publications reported on secondary efficacy outcomes (27 publications), predictors of efficacy (26 publications), other efficacy data (13 publications), or safety outcomes (28 publications) (Table 1).

Only five pooled-trials publications (4.7%) reported a negative conclusion (online Supplementary Table S1). One publication examined the general safety profile of duloxetine, two examined suicidality during paroxetine treatment, one examined discontinuation symptoms after stopping desvenlafaxine, and one examined the effect of desvenlafaxine on blood pressure. All five concluded that the drug was associated with more adverse events than placebo. Fourteen (13.1%) publications had neutral conclusions (predictors of efficacy = 5; other efficacy = 7; safety = 2) while the remaining 88 (82.2%) were positively framed.

Discussion

Main findings

To our knowledge, this study is the first to show that pooled-trials publication bias constitutes a specific form of reporting bias, which results in the publication of many positively framed articles supporting use of a drug, while the original, negative efficacy results of included trials remain obscured. Although 32 of 74 antidepressant trials were not published in stand-alone articles, only nine were completely unpublished, while the other 23 were included in pooled-trials publications. Trials lacking positive results were often exclusively published in pooled-trials publications (41.1%), whereas positive trials were usually published as stand-alone articles (96.3%). Importantly, only 12% of all pooled-trials publications [including 10 (43%) of 23 trials] examined the original primary research question (efficacy compared with placebo), and individual trial results for this original primary question were provided for only 22% of trials published exclusively in pooled-trials publications. Finally, only 5% of pooled-trials publications had a negative conclusion, all of which concerned safety. Therefore, although these trials have technically been published, the negative efficacy results are obscured, thus distorting the drugs’ apparent risk–benefit profile.

For some drugs, particularly duloxetine, the number of pooled-trials publications was very high. Trials HMAT-A and HMAQ-B were included in 44 and 36 publications, respectively. Our study thus replicates a prior report on the ‘salami slicing’ of duloxetine trials (Spielmans et al., 2010). We also found many pooled-trials publications for venlafaxine (18 publications for immediate- and extended-release combined) and escitalopram (16 publications). Several of these publications seemed redundant. For instance, three pooled-trials publications compared the efficacy of escitalopram to citalopram; three examined the effects of age and gender on the efficacy of venlafaxine; and eight compared the efficacy of venlafaxine to selective serotonin reuptake inhibitors.

It is noteworthy that duloxetine (approved in 2004) and escitalopram (approved in 2002) were the two newest antidepressants in Turner et al.’s (2008) meta-analysis, although venlafaxine ER was approved somewhat earlier in 1997. This suggests that the practice of pooling trials in many separate publications is relatively new, perhaps developing concurrently with physicians’ growing skepticism of advertising and sales representatives and greater trust in peer-reviewed publications (Spielmans and Parry, 2010). Consistent with this, a previous study examining antidepressants approved between 1989 and 1994 identified at most six pooled-trials publications for a single drug (Melander et al., 2003). Among novel antidepressants, we also found 12 pooled-trials publications for desvenlafaxine, but none for vilazodone. The latter may be because the unpublished vilazodone trials used a variety of dosages and sometimes used flexible dosages that included the approved dosage (20–40 mg), but also higher (e.g. 20–100 mg) or lower (e.g. 10–20 mg) doses.

Others have also noted that antidepressant meta-analyses (including pooled-trials publications) are massively produced, frequently have some kind of industry involvement, and rarely include negative statements (Ebrahim et al., 2015). In our study, 97 (91%) of 107 pooled-trials publications had at least one industry-employed author and, as noted, only five publications had negative conclusions. In all cases, this was because the antidepressant was associated with more adverse events than placebo, a finding that is not unexpected.

Although we cannot be certain of the intent behind these pooled-trials publications, the fact that they usually included industry employees as authors raises the question as to whether commercial interests may have played some role. In light of the growing concern that the medical literature may function as a marketing tool for pharmaceutical companies (Smith, 2005; Sismondo, 2009; The PLoS Medicine Editors, 2009; Spielmans and Parry, 2010; Vedula et al., 2012), pooled-trials publications may provide an easy and inexpensive way to keep a drug ‘in the spotlight’. Regardless of intent, the overall effect of these pooled-trials publications is to flood the evidence base with articles that encourage use of a drug, while discouraging results from the original trials remain invisible.

Many pooled-trials publications examined safety and tolerability. Because adverse events may occur infrequently, pooling trials can be necessary to achieve sufficient statistical power. The link between antidepressants and suicidality, for instance, was convincingly established only by pooling trials (Hammad et al., 2006; Stone et al., 2009). However, pooling can also mislead. For instance, bupropion SR was only approved at dosages of 300–400 mg/day, but a pooled-trials publication assessing its safety pooled all doses (50–400 mg/day) (Settle et al., 1999). Since adverse events are often dose-dependent, this is likely to paint an overly optimistic picture. Furthermore, there is ongoing concern that meta-analyses of harm outcomes may be particularly threatened by selective outcome reporting (Saini et al., 2014), a concern that is further increased by the possibility of selective trial inclusion in pooled-trials publications (Thaler et al., 2013).

Limitations

Because some publications provided too little information for matching, we may have missed some relevant pooled-trials publications; however, individual trial results are never included in these publications. We specifically excluded 22 pooled-trials publications because it was unclear which trials were included. A second limitation is that we did not count pooled-trials publications that reported individual trial results for the original primary outcome but focused on a secondary research question, as it seemed unlikely that these results would be found by researchers or clinicians interested in the primary outcome. However, there were only five such publications, including three additional trials. Furthermore, most included trials were conducted prior to requirements for trial registration, and in the current study, more recent negative trials were more frequently published as stand-alone articles. However, older antidepressants are still in common use, so the results of these older trials continue to be relevant to clinicians. More subtle biases, such as spin or pooling trials, may also become more important as non-publication of a full trial becomes less common. We also did not conduct a meta-analysis, but focused on the apparent risk–benefit profile, following previous work in which the effect of reporting biases on apparent efficacy was larger than the effects on the meta-analytic effect size. For depression, for instance, the effect size of antidepressants was inflated from 0.31 to 0.41; while only 51% of trials were positive, 94% of published articles were positive (Turner et al., 2008). Finally, we assessed the presence of pooled-trials publication bias in a narrow field. However, such bias would be expected in other fields of medicine, because reporting bias has been found throughout psychiatry (Turner, 2013b; Le Noury et al., 2015; Roest et al., 2015), medicine (McGauran et al., 2010; Hart et al., 2012; Dwan et al., 2013), and science in general (Fanelli, 2012).

Conclusions

Meta-analyses on reporting bias have been criticized by some for excluding pooled-trials publications, but our study shows that these publications are biased toward positive conclusions. As these publications rarely include individual trial results, they appear to serve primarily to heighten the (positive) visibility of a drug, rather than to transparently report negative results. Therefore, inclusion of pooled-trials publications in meta-analyses could lead to biased results, and caution is warranted. Pooled-trials publications may, however, also provide new information, for instance on secondary outcomes (Thaler et al., 2013). Ideally, individual patient data of all trials should be accessible so that biased reporting can no longer hamper the efforts of systematic reviewers. To mitigate the potential for bias, journal editors could also require pooled-trials publications to present individual trial results or reference stand-alone articles for all included trials. Additionally, editors, peer reviewers, and readers should be aware of the potential for bias and redundancy (Spielmans et al., 2010) and perhaps ask whether pooled-trials publications enhance or merely distort and bloat the evidence base.

In summary, the practice of pooling trials distorts the apparent risk–benefit profile of antidepressants by flooding the literature with publications that highlight positive results and obscure negative results. Together with study publication bias, selective outcome reporting, and spin, pooled-trials publication bias is a form of reporting bias that should be taken into account in future research.

Supplementary material

The supplementary material for this article can be found at https://doi.org/10.1017/S0033291718002805

Author contributions

AR, ET, and PJ conceived of the initial idea for the study. All authors contributed to the final design of the study. YV and AR extracted and analyzed the data and wrote a first draft of the manuscript, which was critically revised by all authors. All authors gave final approval of the version to be published.

Financial support

This study was supported by a grant from the Dutch Brain Foundation to Peter de Jonge, Grant number: KS2011(1)-120. The funder had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript

Conflict of interest

None.

Ethical standards

Ethics approval was not required for this work.

References

Dwan, K, Gamble, C, Williamson, PR and Kirkham, JJ (2013) Systematic review of the empirical evidence of study publication bias and outcome reporting bias – an updated review. PLoS ONE 8, e66844.
Ebrahim, S, Bance, S, Athale, A, Malachowski, C and Ioannidis, JPA (2015) Meta-analyses with industry involvement are massively published and report no caveats for antidepressants. Journal of Clinical Epidemiology 70, 155163.
Fanelli, D (2012) Negative results are disappearing from most disciplines and countries. Scientometrics 90, 891904.
Golden, RN, Nemeroff, CB, McSorley, P, Pitts, CD and Dubé, EM (2002) Efficacy and tolerability of controlled-release and immediate-release paroxetine in the treatment of depression. The Journal of Clinical Psychiatry 63, 577584.
Hammad, TA, Laughren, TP and Racoosin, J (2006) Suicidality in pediatric patients treated with antidepressant drugs. Archives of General Psychiatry 63, 332339.
Hart, B, Lundh, A and Bero, L (2012) Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. British Medical Journal 344, d7202.
Higgins, JPT and Green, S (eds) (2011) Cochrane Handbook for Systematic Reviews of Interventions. Version 5. The Cochrane Collaboration. Lilly press release (2008). Available at https://web.archive.org/web/20080202133358/http://newsroom.lilly.com/ReleaseDetail.cfm?ReleaseID=287919.
Le Noury, J, Nardo, JM, Healy, D, Jureidini, J, Raven, M, Tufanaru, C and Abi-Jaoude, E (2015) Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence. British Medical Journal 351, h4320.
McGauran, N, Wieseler, B, Kreis, J, Schüler, Y-B, Kölsch, H and Kaiser, T (2010) Reporting bias in medical research – a narrative review. Trials 11, 37.
Melander, H, Ahlqvist-Rastad, J, Meijer, G and Beermann, B (2003) Evidence b(i)ased medicine – selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. British Medical Journal 326, 15.
Miller, SM, Naylor, GJ, Murtagh, M and Winslow, G (1989) A double-blind comparison of paroxetine and placebo in the treatment of depressed patients in a psychiatric outpatient clinic. Acta Psychiatrica Scandinavica Supplementum 350, 143144.
OHSU Digital Commons (n.d.). FDA Drug Approval Documents. Available at https://digitalcommons.ohsu.edu/fdadrug/
Rising, K, Bacchetti, P and Bero, L (2008) Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation. PLoS Medicine 5, 15611570.
Roest, AM, de Jonge, P, Williams, CD, de Vries, YA, Schoevers, RA and Turner, EH (2015) Reporting bias in clinical trials investigating the efficacy of second-generation antidepressants in the treatment of anxiety disorders: a report of 2 meta-analyses. JAMA Psychiatry 72, 500510.
Saini, P, Loke, YK, Gamble, C, Altman, DG, Williamson, PR and Kirkham, JJ (2014) Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. British Medical Journal 349, g6501.
Settle, EC, Stahl, SM, Batey, SR, Johnston, JA and Ascher, JA (1999) Safety profile of sustained-release bupropion in depression: results of three clinical trials. Clinical Therapeutics 21, 454463.
Sismondo, S (2009) Ghosts in the machine: publication planning in the medical sciences. Social Studies of Science 39, 171198.
Smith, R (2005) Medical journals are an extension of the marketing arm of pharmaceutical companies. PLoS Medicine 2, e138.
Spielmans, GI and Parry, PI (2010) From evidence-based medicine to marketing-based medicine: evidence from internal industry documents. Journal of Bioethical Inquiry 7, 1329.
Spielmans, GI, Biehn, TL and Sawrey, DL (2010) A case study of salami slicing: pooled analyses of duloxetine for depression. Psychotherapy and Psychosomatics 79, 97106.
Spielmans, GI, Olson, S and Keicher, RM (2017) ‘Salami slicing’ in pooled analyses of second-generation antipsychotics for the treatment of depression. Psychotherapy and Psychosomatics 55108, 171172.
Stone, M, Laughren, TP, Jones, ML, Levenson, M, Holland, PC, Hughes, A, Hammad, TA, Temple, R and Rochester, G (2009) Risk of suicidality in clinical trials of antidepressants in adults: analysis of proprietary data submitted to US Food and Drug Administration. British Medical Journal 339, b2880.
Thaler, KJ, Morgan, LC, Van Noord, M, Jonas, DE, McDonagh, MS, Peterson, K, Glechner, A and Gartlehner, G (2013) A case study of pooled-studies publications indicated potential for both valuable information and bias. Journal of Clinical Epidemiology 66, 10821092.
The PLoS Medicine Editors (2009) Ghostwriting: the dirty little secret of medical publishing that just got bigger. PLoS Medicine 6, e1000156.
Turner, EH (2004) A taxpayer-funded clinical trials registry and results database. PLoS Medicine 1, e60.
Turner, EH (2013 a) How to access and process FDA drug approval packages for use in research. British Medical Journal 347, f5992.
Turner, EH (2013 b) Publication bias, with a focus on psychiatry: causes and solutions. CNS Drugs 27, 457468.
Turner, EH, Matthews, AM, Linardatos, E, Tell, RA and Rosenthal, R (2008) Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine 358, 252260.
Turner, EH, Knoepflmacher, D and Shapley, L (2012) Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database. PLoS Medicine 9, e1001189.
Vedula, SS, Goldman, PS, Rona, IJ, Greene, TM and Dickersin, K (2012) Implementation of a publication strategy in the context of reporting biases. A case study based on new documents from Neurontin litigation. Trials 13, 113.