Skip to main content Accessibility help

Inadequate reporting of trials compromises the applicability of systematic reviews

  • Gerald Gartlehner (a1), Patricia Thieda (a2), Richard A. Hansen (a2), Laura C. Morgan (a2), Janelle A. Shumate (a3) and Daniel B. Nissman (a4)...


Background: Uncertainty about the applicability of controlled trial findings is an increasing concern for clinicians and policy decision makers. This study aimed to determine whether information reported in studies included in systematic reviews was adequate enough to assess their applicability.

Methods: We used the databases of four recently conducted systematic reviews on the comparative efficacy and safety of second-generation antidepressants, inhaled corticosteroids, Alzheimer's drugs, and targeted immune modulators. We developed and pilot-tested a questionnaire to assess the adequacy of reporting with respect to seven previously validated criteria of study design that distinguish explanatory from pragmatic studies. For each of the 137 included studies, two reviewers independently assessed the adequacy of reporting.

Results: Overall, only 12 percent of the included studies provided sufficient information to reliably distinguish explanatory from pragmatic studies. The areas with the greatest lack of reporting were the setting of the study, methods of adverse event assessment, and sample size considerations to determine a minimally important difference from a patient perspective.

Conclusions: Substantial shortcomings in reporting exist in aspects of study design important to determine whether a study is applicable to specific populations of interest.



Hide All
1. Agency for Healthcare Research and Quality (AHRQ). (accessed November 15, 2008).
2. Altman, DG, Schulz, KF, Moher, D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001;134:663694.
3. Atkins, D. Creating and synthesizing evidence with decision makers in mind: integrating evidence from clinical trials and other study designs. Med Care. 2007;45 (Suppl 2):S16S22.
4. Chan, AW, Hrobjartsson, A, Haahr, MT, et al. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291:24572465.
5. Dumville, JC, Torgerson, DJ, Hewitt, CE. Reporting attrition in randomized controlled trials. BMJ. 2006;332:969971.
6. Ethgen, M, Boutron, I, Baron, G, et al. Reporting of harm in randomized, controlled trials of nonpharmacologic treatment for rheumatic disease. Ann Intern Med. 2005;143:2025.
7. Fortin, M, Dionne, J, Pinho, G, et al. Randomized controlled trials: do they have external validity for patients with multiple comorbidities? Ann Fam Med. 2006;4:104108.
8. Gartlehner, G, Hansen, RA, Kahwati, L, et al. Drug class review on second generation antidepressants. (accessed November 15, 2008).
9. Gartlehner, G, Hansen, RA, Nissman, D, et al. A simple and valid tool distinguished efficacy from effectiveness studies. J Clin Epidemiol. 2006;59:10401048.
10. Gartlehner, G, Hansen, RA, Thieda, P, et al. Drug class review on targeted immune modulators. (accessed November 15, 2008).
11. Godwin, M, Ruhland, L, Casson, I, et al. Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity. BMC Med Res Methodol. 2003;3:28.
12. Gross, CP, Mallory, R, Heiat, A, et al. Reporting the recruitment process in clinical trials: who are these patients and how did they get there? Ann Intern Med. 2002;137:1016.
13. Hansen, RA, Gartlehner, G, Kaufer, DJ, et al. Drug class review on Alzheimer's drugs. (accessed November 15, 2008).
14. Hansen, RA, Gartlehner, G, Lohr, KN, et al. Drug class review on inhaled corticosteroids. (accessed November 15, 2008).
15. Haynes, B. Can it work? Does it work? Is it worth it? The testing of healthcare interventions is evolving. BMJ. 1999;319:652653.
16. Hazell, L, Shakir, SA. Under-reporting of adverse drug reactions: a systematic review. Drug Saf. 2006;29:385396.
17. Hewitt, C, Hahn, S, Torgerson, DJ, et al. Adequacy and reporting of allocation concealment: review of recent trials published in four general medical journals. BMJ. 2005;330:10571058.
18. Huwiler-Muntener, K, Juni, P, Junker, C, et al. Quality of reporting of randomized trials as a measure of methodological quality. JAMA. 2002;287:28012804.
19. Ioannidis, JP, Evans, SJ, Gotzsche, PC, et al. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med. 2004;141:781788.
20. MacRae, KD. Pragmatic versus explanatory trials. Int J Technol Assess Health Care. 1989;5:333339.
21. Malmivaara, A, Koes, BW, Bouter, LM, et al. Applicability and clinical relevance of results in randomized controlled trials: the Cochrane review on exercise therapy for low back pain as an example. Spine. 2006;31:14051409.
22. Moher, D, Cook, DJ, Eastwood, S, et al. Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Onkologie. 2000;23:597602.
23. Moher, D, Schulz, KF, Altman, D. The CONSORT Statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials 2001. Explore (NY). 2005;1:4045.
24. Mullner, M, Matthews, H, Altman, DG. Reporting on statistical methods to adjust for confounding: a cross-sectional survey. Ann Intern Med. 2002;136:122126.
25. Mulrow, CD. Rationale for systematic reviews. BMJ. 1994;309:597599.
26. Oregon Health and Science University (OHSU). (accessed November 15, 2008).
27. Plint, AC, Moher, D, Morrison, A, et al. Does the CONSORT checklist improve the quality of reports of randomized controlled trials? A systematic review. Med J Aust. 2006;185:263267.
28. Schwartz, D, Lellouch, J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967;20:637648.
29. Slutsky, J, Atkins, D, Chang, S, et al. Comparing medical interventions: AHRQ and the effective health-care program. J Clin Epidemiol. 2008 [Epub ahead of print].
30. Sokka, T, Pincus, T. Eligibility of patients in routine care for major clinical trials of anti-tumor necrosis factor alpha agents in rheumatoid arthritis. Arthritis Rheum. 2003;48:313318.
31. Zwarenstein, M, Treweek, S, Gagnier, JJ, et al. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337:a2390.


Inadequate reporting of trials compromises the applicability of systematic reviews

  • Gerald Gartlehner (a1), Patricia Thieda (a2), Richard A. Hansen (a2), Laura C. Morgan (a2), Janelle A. Shumate (a3) and Daniel B. Nissman (a4)...


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed