Skip to main content Accessibility help

Methodological quality of systematic reviews on treatments for depression: a cross-sectional study

  • V. C. H. Chung (a1) (a2), X. Y. Wu (a2) (a3), Y. Feng (a1), R. S. T. Ho (a1), S. Y. S. Wong (a1) and D. Threapleton (a1)...



Depression is one of the most common mental disorders and identifying effective treatment strategies is crucial for the control of depression. Well-conducted systematic reviews (SRs) and meta-analyses can provide the best evidence for supporting treatment decision-making. Nevertheless, the trustworthiness of conclusions can be limited by lack of methodological rigour. This study aims to assess the methodological quality of a representative sample of SRs on depression treatments.


A cross-sectional study on the bibliographical and methodological characteristics of SRs published on depression treatments trials was conducted. Two electronic databases (the Cochrane Database of Systematic Reviews and the Database of Abstracts of Reviews of Effects) were searched for potential SRs. SRs with at least one meta-analysis on the effects of depression treatments were considered eligible. The methodological quality of included SRs was assessed using the validated AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool. The associations between bibliographical characteristics and scoring on AMSTAR items were analysed using logistic regression analysis.


A total of 358 SRs were included and appraised. Over half of included SRs (n = 195) focused on non-pharmacological treatments and harms were reported in 45.5% (n = 163) of all studies. Studies varied in methods and reporting practices: only 112 (31.3%) took the risk of bias among primary studies into account when formulating conclusions; 245 (68.4%) did not fully declare conflict of interests; 93 (26.0%) reported an ‘a priori’ design and 104 (29.1%) provided lists of both included and excluded studies. Results from regression analyses showed: more recent publications were more likely to report ‘a priori’ designs [adjusted odds ratio (AOR) 1.31, 95% confidence interval (CI) 1.09–1.57], to describe study characteristics fully (AOR 1.16, 95% CI 1.06–1.28), and to assess presence of publication bias (AOR 1.13, 95% CI 1.06–1.19), but were less likely to list both included and excluded studies (AOR 0.86, 95% CI 0.81–0.92). SRs published in journals with higher impact factor (AOR 1.14, 95% CI 1.04–1.25), completed by more review authors (AOR 1.12, 95% CI 1.01–1.24) and SRs on non-pharmacological treatments (AOR 1.62, 95% CI 1.01–2.59) were associated with better performance in publication bias assessment.


The methodological quality of included SRs is disappointing. Future SRs should strive to improve rigour by considering of risk of bias when formulating conclusions, reporting conflict of interests and authors should explicitly describe harms. SR authors should also use appropriate methods to combine the results, prevent language and publication biases, and ensure timely updates.


Corresponding author

*Address for correspondence: X. Y. Wu, Room 509, JC School of Public Health Building, Prince of Wales Hospital, Shatin, NT, Hong Kong. (Email:


Hide All

These authors contributed equally to this paper.



Hide All
American Psychiatric Association (1980). Diagnostical and Statistical Manual of Mental Disorders (DSM-III). American Psychiatric Association: Washington, DC.
Burda, BU, Holmer, HK, Norris, SL (2016). Limitations of a measurement tool to assess systematic reviews (AMSTAR) and suggestions for improvement. Systematic Reviews 5, 58.
Busse, JW, Bruno, P, Malik, K, Connell, G, Torrance, D, Ngo, T, Kirmayr, K, Avrahami, D, Riva, JJ, Ebrahim, S, Struijs, PA, Brunarski, D, Burnie, SJ, LeBlanc, F, Coomes, EA, Steenstra, IA, Slack, T, Rodine, R, Jim, J, Montori, VM, Guyatt, GH (2014). An efficient strategy allowed English-speaking reviewers to identify foreign-language articles eligible for a systematic review. Journal of Clinical Epidemiology 67, 547553.
Chung, VCH, Ho, RST, Wu, XY, Fung, DHY, Lai, X, Wu, JCW, Wong, SYS (2015). Are meta-analyses of Chinese herbal medicine trials trustworthy and clinically applicable? A cross-sectional study. Journal of Ethnopharmacology 162, 4754.
Cipriani, A, Barbui, C, Butler, R, Hatcher, S, Geddes, J (2011). Depression in adults: drug and physical treatments. BMJ Clinical Evidence 2011, 1003.
Cochrane Community (beta) (2015). Database of Abstracts of Reviews of Effects (DARE).
Cohen, JF, Korevaar, DA, Wang, J, Spijker, R, Bossuyt, PM (2015). Should we search Chinese biomedical databases when performing systematic reviews? Systematic Reviews 4, 23.
Dwan, K, Altman, DG, Arnaiz, JA, Bloom, J, Chan, AW, Cronin, E, Decullier, E, Easterbrook, PJ, Von Elm, E, Gamble, C, Ghersi, D, Ioannidis, JPA, Simes, J, Williamson, PR (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3, e3081.
Egger, M, Zellweger-Zähner, T, Schneider, M, Junker, C, Lengeler, C, Antes, G (1997). Language bias in randomised controlled trials published in English and German. Lancet 350, 326329.
Egger, M, Smith, GD, Sterne, JA (2001). Uses and abuses of meta-analysis. Clinical Medicine 1, 478484.
Fleming, PS, Koletsi, D, Seehra, J, Pandis, N (2014). Systematic reviews published in higher impact clinical journals were of higher quality. Journal of Clinical Epidemiology 67, 754759.
Hamilton, M (1960). A rating scale for depression. Journal of Neurology, Neurosurgery, and Psychiatry 23, 5662.
Higgins, J, Green, S (2011). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration. Available from
Ho, RS, Wu, XY, Yuan, JQ, Liu, SY, Lai, X, Wong, SY, Chung, VC (2015). Methodological quality of meta-analyses on treatments for chronic obstructive pulmonary disease: a cross-sectional study using the AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool. Npj Primary Care Respiratory Medicine 25, 14102.
Jørgensen, AW, Hilden, J, Gøtzsche, PC (2006). Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ 333, 782.
Jüni, P, Holenstein, F, Sterne, J, Bartlett, C, Egger, M (2002). Direction and impact of language bias in meta-analyses of controlled trials: empirical study. International Journal of Epidemiology 31, 115123.
Lexchin, J, Bero, LA, Djulbegovic, B, Clark, O (2003). Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 326, 11671170.
Moher, D, Tsertsvadze, A (2006). Systematic reviews: when is an update an update? Lancet 367, 881883.
Moher, D, Tetzlaff, J, Tricco, AC, Sampson, M, Altman, DG (2007). Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 4, e78.
Moher, D, Liberati, A, Tetzlaff, J, Altman, DG (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339, b2535.
Moja, LP, Telaro, E, D'Amico, R, Moschetti, I, Coe, L, Liberati, A (2005). Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ 330, 1053.
Morrison, A, Polisena, J, Husereau, D, Moulton, K, Clark, M, Fiander, M, Mierzwinski-Urban, M, Clifford, T, Hutton, B, Rabb, D (2012). The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. International Journal of Technology Assessment in Health Care 28, 138144.
Mulrow, CD, Cook, DJ, Davidoff, F (1997). Systematic reviews: critical links in the great chain of evidence. Annals of Internal Medicine 126, 389391.
Mykletun, A, Bjerkeset, O, Overland, S, Prince, M, Dewey, M, Stewart, R (2009). Levels of anxiety and depression as predictors of mortality: the HUNT study. British Journal of Psychiatry 195, 118125.
Page, MJ, Shamseer, L, Altman, DG, Tetzlaff, J, Sampson, M, Tricco, AC, Catalá-López, F, Li, L, Reid, EK, Sarkis-Onofre, R, Moher, D (2016). Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Medicine 13, e1002028.
Papageorgiou, SN, Papadopoulos, MA, Athanasiou, AE (2011). Evaluation of methodology and quality characteristics of systematic reviews in orthodontics. Orthodontics and Craniofacial Research 14, 116137.
Remschmidt, C, Wichmann, O, Harder, T (2014). Methodological quality of systematic reviews on influenza vaccination. Vaccine 32, 16781684.
Sequeira-Byron, P, Fedorowicz, Z, Jagannath, VA, Sharif, MO (2011). An AMSTAR assessment of the methodological quality of systematic reviews of oral healthcare interventions published in the Journal of Applied Oral Science (JAOS). Journal of Applied Oral Science 19, 440447.
Shea, B, Grimshaw, J, Wells, G, Boers, M, Andersson, N, Hamel, C, Porter, A, Tugwell, P, Moher, D, Bouter, L (2007). Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology 7, 10.
Shea, B, Hamel, C, Wells, GA, Bouter, L, Kristjansson, E, Grimshaw, J, Henry, D, Boers, M (2009). AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology 62, 10131020.
Wegewitz, U, Weikert, B, Fishta, A, Jacobs, A, Pieper, D (2016). Resuming the discussion of AMSTAR: what can (should) be made better? BMC Medical Research Methodology 16, 111.
World Health Organization (2012). Depression.
Wu, XY, Tang, JL, Mao, C, Yuan, JQ, Qin, Y, Chung, VCH (2013). Systematic reviews and meta-analyses of traditional Chinese medicine must search Chinese databases to reduce language bias. Evidence-based Complementary and Alternative Medicine: eCAM 2013, 812179.
Wu, XY, Du, XJ, Ho, RS, Lee, CC, Yip, BH, Wong, MC, Wong, SY, Chung, VC (2016 a). Characteristics and methodological quality of meta-analyses on hypertension treatments-a cross-sectional study. Journal of Clinical Hypertension 19, 137142.
Wu, XY, Lam, VC, Yu, YF, Ho, RS, Feng, Y, Wong, CH, Yip, BH, Tsoi, K, Wong, SY, Chung, VC (2016 b). Epidemiological characteristics and methodological quality of meta-analyses on diabetes mellitus treatment: a systematic review. European Journal of Endocrinology 175, 353360.


Type Description Title
Supplementary materials

Chung supplementary material
Chung supplementary material

 Word (62 KB)
62 KB


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed