Skip to main content Accessibility help

The role of meta-analyses and umbrella reviews in assessing the harms of psychotropic medications: beyond qualitative synthesis

  • M. Solmi (a1) (a2) (a3), C. U. Correll (a4) (a5) (a6), A. F. Carvalho (a7) (a8) and J. P. A. Ioannidis (a9) (a10) (a11) (a12)


ὠφελέειν, ἢ μὴ βλάπτειν (Primum non nocere) – Hιppocrates’ principle should still guide daily medical prescribing. Therefore, assessing evidence of psychopharmacologic agents’ safety and harms is essential. Randomised controlled trials (RCTs) and observational studies may provide complementary information about harms of psychopharmacologic medications from both experimental and real-world settings. It is considered that RCTs provide a better control of confounding variables, while observational studies provide evidence from larger samples, longer follow-ups, in more representative samples, which may be more reflective of real-life clinical scenarios. However, this may not always hold true. Moreover, in observational studies, safety data are poorly or inconsistently reported, precluding reliable quantitative synthesis in meta-analyses. Beyond individual studies, meta-analyses, which represent the highest level of ‘evidence’, can be misleading, redundant and of low methodological quality. Overlapping meta-analyses sometimes even reach different conclusions on the same topic. Meta-analyses should be assessed systematically. Descriptive reviews of reviews can be poorly informative. Conversely, ‘umbrella reviews’ can use a quantitative approach to grade evidence. In this editorial, we present the main factors involved in the assessment of psychopharmacologic agents’ harms from individual studies, meta-analyses and umbrella reviews. Study design features, sample size, number of the events of interest, summary effect sizes, p-values, heterogeneity, 95% prediction intervals, confounding factor adjustment and tests of bias (e.g., small-study effects and excess significance) can be combined with other assessment tools, such as AMSTAR and GRADE to create a framework for assessing the credibility of evidence.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      The role of meta-analyses and umbrella reviews in assessing the harms of psychotropic medications: beyond qualitative synthesis
      Available formats

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      The role of meta-analyses and umbrella reviews in assessing the harms of psychotropic medications: beyond qualitative synthesis
      Available formats

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      The role of meta-analyses and umbrella reviews in assessing the harms of psychotropic medications: beyond qualitative synthesis
      Available formats


Corresponding author

Author for correspondence: Marco Solmi, E-mail:


Hide All
American Psychiatric Association (2013) Diagnostic and Statistical Manual of Mental Disorders, 5th Edn. Washington, DC: Author.
Belbasis, L, Bellou, V and Evangelou, E (2016) Environmental risk factors and amyotrophic lateral sclerosis: an Umbrella review and critical assessment of current evidence from systematic reviews and meta-analyses of observational studies. Neuroepidemiology 46, 96105.
Bellou, V, Belbasis, L, Tzoulaki, I, Evangelou, E and Ioannidis, JP (2016) Environmental risk factors and Parkinson's disease: an umbrella review of meta-analyses. Parkinsonism Related Disorders 23, 19.
Cipriani, A, Furukawa, TA, Salanti, G, Chaimani, A, Atkinson, LZ, Ogawa, Y, Leucht, S, Ruhe, HG, Turner, EH, Higgins, JPT, Egger, M, Takeshima, N, Hayasaka, Y, Imai, H, Shinohara, K, Tajika, A, Ioannidis, JPA and Geddes, JR (2018) Comparative efficacy and acceptability of 21 antidepressant drugs for the acute treatment of adults with major depressive disorder: a systematic review and network meta-analysis. Lancet 391, 13571366.
Cohen, J (1988) Statistical Power Analysis for the Behavioral Sciences. (Ed. Routledge). ISBN 1-134-74270-3.
Correll, CU, Rubio, JM, Inczedy-Farkas, G, Birnbaum, ML, Kane, JM and Leucht, S (2017) Efficacy of 42 pharmacologic cotreatment strategies added to antipsychotic monotherapy in schizophrenia: systematic overview and quality appraisal of the meta-analytic evidence. Journal of American Medical Association Psychiatry 74, 675684.
Ebrahim, S, Sohani, ZN, Montoya, L, Agarwal, A, Thorlund, K, Mills, EJ and Ioannidis, JP (2014) Reanalyses of randomized clinical trial data. Journal of America Medical Association 312, 10241032.
Egger, M, Davey Smith, G, Schneider, M and Minder, C (1997) Bias in meta-analysis detected by a simple, graphical test. British Medical Journal 315, 629634.
Higgins, JP and Thompson, SG (2002) Quantifying heterogeneity in a meta-analysis. Statistics in Medicine 21, 15391558.
IntHout, J, Ioannidis, JP, Rovers, MM and Goeman, JJ (2016) Plea for routinely presenting prediction intervals in meta-analysis. British Medical Journal Open 6, e010247.
Ioannidis, JP (2009 a) Adverse events in randomized trials: neglected, restricted, distorted, and silenced. Archives of Internal Medicine 169, 17371739.
Ioannidis, JP (2009 b) Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. Canadian Medical Association Journal 181, 488493.
Ioannidis, JP (2016) The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. The Milbank Quarterly 94, 485514.
Ioannidis, J (2017) Next-generation systematic reviews: prospective meta-analysis, individual-level data, networks and umbrella reviews. British Journal of Sports Medicine 51, 14561458.
Ioannidis, JPA (2018) The proposal to lower p value thresholds to .005. Journal of American Medical Association 319, 14291430.
Ioannidis, JP and Lau, J (1998) Can quality of clinical trials and meta-analyses be quantified? Lancet 352, 590591.
Ioannidis, JP and Trikalinos, TA (2007) An exploratory test for an excess of significant findings. Clinical Trials 4, 245253..
Ioannidis, JP, Evans, SJ, Gotzsche, PC, O'Neill, RT, Altman, DG, Schulz, K, Moher, D and Group, C (2004) Better reporting of harms in randomized trials: an extension of the CONSORT statement. Annals of Internal Medicine 141, 781788.
Ioannidis, JP, Patsopoulos, NA and Evangelou, E (2007) Uncertainty in heterogeneity estimates in meta-analyses. British Medical Journal 335, 914916.
Iqbal, SA, Wallach, JD, Khoury, MJ, Schully, SD and Ioannidis, JP (2016) Reproducible research practices and transparency across the biomedical literature. PLoS Biology 14, e1002333.
Lau, J, Ioannidis, JP, Terrin, N, Schmid, CH and Olkin, I (2006) The case of the misleading funnel plot. British Medical Journal 333, 597600.
Leucht, S, Cipriani, A, Spineli, L, Mavridis, D, Orey, D, Richter, F, Samara, M, Barbui, C, Engel, RR, Geddes, JR, Kissling, W, Stapf, MP, Lassig, B, Salanti, G and Davis, JM (2013) Comparative efficacy and tolerability of 15 antipsychotic drugs in schizophrenia: a multiple-treatments meta-analysis. Lancet 382, 951962.
Li, X, Meng, X, Timofeeva, M, Tzoulaki, I, Tsilidis, KK, Ioannidis, JP, Campbell, H and Theodoratou, E (2017) Serum uric acid levels and multiple health outcomes: umbrella review of evidence from observational studies, randomised controlled trials, and Mendelian randomisation studies. British Medical Journal 357, j2376.
Munafò, MR, Nosek, BA, Bishop, DVM, Button, KS, Chambers, CD, Percie du Sert, N, Simonsohn, U, Wagenmakers, E-J, Ware, JJ and Ioannidis, JPA (2017) A manifesto for reproducible science. Nature Human Behaviour 1, 0021.
Naudet, F, Sakarovitch, C, Janiaud, P, Cristea, I, Fanelli, D, Moher, D and Ioannidis, JPA (2018) Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS medicine. British Medical Journal 360, k400.
Papanikolaou, PN and Ioannidis, JP (2004) Availability of large-scale evidence on specific harms from systematic reviews of randomized trials. American Journal of Medicine 117, 582589.
Papanikolaou, PN, Christidi, GD and Ioannidis, JP (2006) Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies. Canadian Medical Association Journal 174, 635641.
Pollock, M, Fernandes, RM and Hartling, L (2017) Evaluation of AMSTAR to assess the methodological quality of systematic reviews in overviews of reviews of healthcare interventions. BioMed Central Medical Research Methodology 17, 48.
Prasad, V and Jena, AB (2013) Prespecified falsification end points: can they validate true observational associations? Journal of American Medical Association 309, 241242.
Sawilowsky, S (2009) New effect size rules of thumb. Journal of Modern Applied Statistical Methods 8, 467474.
Schünemann, H, Brożek, J, Guyatt, G and Oxman, A (2013) Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach. Available at (Accessed 31 May 2018).
Shea, BJ, Hamel, C, Wells, GA, Bouter, LM, Kristjansson, E, Grimshaw, J, Henry, DA and Boers, M (2009) AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiol 62, 10131020.
Shea, BJ, Reeves, BC, Wells, G, Thuku, M, Hamel, C, Moran, J, Moher, D, Tugwell, P, Welch, V, Kristjansson, E and Henry, DA (2017) AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. British Medical Journal 358, j4008.
Solmi, M, Murru, A, Pacchiarotti, I, Undurraga, J, Veronese, N, Fornaro, M, Stubbs, B, Monaco, F, Vieta, E, Seeman, MV, Correll, CU and Carvalho, AF (2017) Safety, tolerability, and risks associated with first- and second-generation antipsychotics: a state-of-the-art clinical review. Therapeutics and clinical risk management. 13, 757777.
Sterne, JA, Sutton, AJ, Ioannidis, JP, Terrin, N, Jones, DR, Lau, J, Carpenter, J, Rucker, G, Harbord, RM, Schmid, CH, Tetzlaff, J, Deeks, JJ, Peters, J, Macaskill, P, Schwarzer, G, Duval, S, Altman, DG, Moher, D and Higgins, JP (2011) Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. British Medical Journal 343, d4002.
Stodden, V, McNutt, M, Bailey, DH, Deelman, E, Gil, Y, Hanson, B, Heroux, MA, Ioannidis, JP and Taufer, M (2016) Enhancing reproducibility for computational methods. Science 354, 12401241.
Szucs, D and Ioannidis, JPA (2017) When null hypothesis significance testing is unsuitable for research: a reassessment. Frontiers in Human Neuroscience 11, 390.
Theodoratou, E, Tzoulaki, I, Zgaga, L and Ioannidis, JP (2014) Vitamin D and multiple health outcomes: umbrella review of systematic reviews and meta-analyses of observational studies and randomised trials. British Medical Journal 348, g2035.
Veronese, N, Solmi, M, Caruso, MG, Giannelli, G, Osella, AR, Evangelou, E, Maggi, S, Fontana, L, Stubbs, B and Tzoulaki, I (2018) Dietary fiber and health outcomes: an umbrella review of systematic reviews and meta-analyses. American Journal of Clinical Nutrition 107, 436444.
Wasserstein, RL and Lazar, NA (2016) The ASA's statement on p-values: context, process, and purpose. The American Statistician 70, 129133.
Weihrauch, TR and Gauler, TC (1999) Placebo-efficacy and adverse effects in controlled clinical trials. Arzneimittel-Forschung 49, 385393.
Wells, G, Shea, B, O'Connell, D, Peterson, J, Welch, V, Losos, M and Tugwell, P (2013) The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Available at (Accessed 31 May 2018).
Zorzela, L, Loke, YK, Ioannidis, JP, Golder, S, Santaguida, P, Altman, DG, Moher, D, Vohra, S and Group, PR (2016) PRISMA harms checklist: improving harms reporting in systematic reviews. British Medical Journal 352, i157.



Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed