Skip to main content Accessibility help
×
Home

Pre-post effect sizes should be avoided in meta-analyses

  • P. Cuijpers (a1) (a2), E. Weitz (a1) (a2), I. A. Cristea (a3) and J. Twisk (a2)

Abstract

Aims

The standardised mean difference (SMD) is one of the most used effect sizes to indicate the effects of treatments. It indicates the difference between a treatment and comparison group after treatment has ended, in terms of standard deviations. Some meta-analyses, including several highly cited and influential ones, use the pre-post SMD, indicating the difference between baseline and post-test within one (treatment group).

Methods

In this paper, we argue that these pre-post SMDs should be avoided in meta-analyses and we describe the arguments why pre-post SMDs can result in biased outcomes.

Results

One important reason why pre-post SMDs should be avoided is that the scores on baseline and post-test are not independent of each other. The value for the correlation should be used in the calculation of the SMD, while this value is typically not known. We used data from an ‘individual patient data’ meta-analysis of trials comparing cognitive behaviour therapy and anti-depressive medication, to show that this problem can lead to considerable errors in the estimation of the SMDs. Another even more important reason why pre-post SMDs should be avoided in meta-analyses is that they are influenced by natural processes and characteristics of the patients and settings, and these cannot be discerned from the effects of the intervention. Between-group SMDs are much better because they control for such variables and these variables only affect the between group SMD when they are related to the effects of the intervention.

Conclusions

We conclude that pre-post SMDs should be avoided in meta-analyses as using them probably results in biased outcomes.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Pre-post effect sizes should be avoided in meta-analyses
      Available formats
      ×

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Pre-post effect sizes should be avoided in meta-analyses
      Available formats
      ×

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Pre-post effect sizes should be avoided in meta-analyses
      Available formats
      ×

Copyright

Corresponding author

*Address for correspondence: Professor P. Cuijpers, Ph.D., Department of Clinical, Neuro and Developmental Psychology, Vrije Universiteit Amsterdam, Van der Boechorststraat 1, 1081 BT Amsterdam, The Netherlands. (Email: p.cuijpers@vu.nl)

References

Hide All
Balk, EM, Earley, A, Patel, K, Trikalinos, TA, Dahabreh, IJ (2012). Empirical Assessment of Within-arm Correlation Imputation in Trials of Continuous Outcomes (Methods Research Report No. AHRQ Publication No. 12(13)-EHC141-EF). Agency for Healthcare Research and Quality: Rockville, MD. Retrieved from www.effectivehealthcare.ahrq.gov/reports/final.cfm
Beck, AT, Ward, CH, Mendelson, M, Mock, J, Erbaugh, J (1961). An inventory for measuring depression. Archives of General Psychiatry 4, 561571.
Beck, AT, Steer, RA, Brown, GK (1996). BDI-II. Beck Depression Inventory, 2nd edn. Manual. Psychological Corporation: San Antonio.
Biostat Inc (2015). Comprehensive Meta-Analysis (Version 2.2.064). Retrieved from www.meta-analysis.com
Cuijpers, P, Karyotaki, E, Weitz, E, Andersson, G, Hollon, SD, van Straten, A (2014 a). The effects of psychotherapies for major depression in adults on remission, recovery and improvement: a meta-analysis. Journal of Affective Disorders 159, 118126.
Cuijpers, P, Weitz, E, Twisk, J, Kuehner, C, Cristea, I, David, D, Hollon, SD (2014 b). Gender as predictor and moderator of outcome in cognitive behavior therapy and pharmacotherapy for adult depression: an ‘individual patient data’ meta-analysis. Depression and Anxiety 31, 941951.
Hamilton, M (1960). A rating scale for depression. Journal of Neurology, Neurosurgery, and Psychiatry 23, 5662.
Hesser, H, Weise, C, Westin, VZ, Andersson, G (2011). A systematic review and meta-analysis of randomized controlled trials of cognitive–behavioral therapy for tinnitus distress. Clinical Psychology Review 31, 545553.
Higgins, JPT, Green, S (2011). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration: Chicester, England.
Hofmann, SG, Wu, JQ, Boettcher, H (2014). Effect of cognitive-behavioral therapy for anxiety disorders on quality of life: a meta-analysis. Journal of Consulting and Clinical Psychology 82, 375391.
Johnsen, TJ, Friborg, O (2015). The effects of cognitive behavioral therapy as an anti-depressive treatment is falling: a meta-analysis. Psychological Bulletin, Online First Publication 141, 747768.
Kirsch, I, Deacon, BJ, Huedo-Medina, TB, Scoboria, A, Moore, TJ, Johnson, BT (2008). Initial severity and antidepressant benefits: a meta-analysis of data submitted to the Food and Drug Administration. PLoS Medicine 5, e45.
Leichsenring, F, Rabung, S (2008). Effectiveness of long-term psychodynamic psychotherapy: a meta-analysis. JAMA 300, 15511565.
van der Lem, R, de Wever, WW, van der Wee, NJ, van Veen, T, Cuijpers, P, Zitman, FG (2012 a). The generalizability of psychotherapy efficacy trials in major depressive disorder: an analysis of the influence of patient selection in efficacy trials on symptom outcome in daily practice. BMC Psychiatry 12, 192.
van der Lem, R, van der Wee, NJA, van Veen, T, Zitman, FG (2012 b). Efficacy versus effectiveness: a direct comparison of the outcome of treatment for mild to moderate depression in randomized controlled trials and daily practice. Psychotherapy and Psychosomatics 81, 226234.
Vittengl, JR, Jarrett, RB, Weitz, E, Hollon, SD, Twisk, J, Cristea, I, Cuijpers, P (2016). Divergent outcomes in cognitive-behavioral therapy and pharmacotherapy for adult depression. American Journal of Psychiatry 173, 481490.
Weitz, ES, Hollon, SD, Twisk, J, van Straten, A, Huibers, MJ, David, D, Cuijpers, P (2015). Baseline depression severity as moderator of depression outcomes between cognitive behavioral therapy vs pharmacotherapy: an individual patient data meta-analysis. JAMA Psychiatry 72, 11021109.

Keywords

Pre-post effect sizes should be avoided in meta-analyses

  • P. Cuijpers (a1) (a2), E. Weitz (a1) (a2), I. A. Cristea (a3) and J. Twisk (a2)

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed