Skip to main content Accessibility help
×
Home

Is grey literature essential for a better control of publication bias in psychiatry? An example from three meta-analyses of schizophrenia

Published online by Cambridge University Press:  16 April 2020

José Luis R. Martin
Affiliation:
Department of Clinical Research, Foundation for Health Research in Castilla-La Mancha (FISCAM), Edificio Bulevar, C/Berna, No. 2, Local 0-2, 45003– Toledo, Castilla-La Mancha, Spain
Víctor Pérez
Affiliation:
Department of Psychiatry, Hospital de la Santa Creu i Sant Pau, Barcelona, Spain
Montse Sacristán
Affiliation:
Department of Clinical Research, Foundation for Health Research in Castilla-La Mancha (FISCAM), Edificio Bulevar, C/Berna, No. 2, Local 0-2, 45003– Toledo, Castilla-La Mancha, Spain
Enric Álvarez
Affiliation:
Department of Psychiatry, Hospital de la Santa Creu i Sant Pau, Barcelona, Spain
Corresponding
E-mail address:

Abstract

Systematic reviews in mental health have become useful tools for health professionals in view of the massive amount and heterogeneous nature of biomedical information available today. In order to determine the risk of bias in the studies evaluated and to avoid bias in generalizing conclusions from the reviews it is therefore important to use a very strict methodology in systematic reviews. One bias which may affect the generalization of results is publication bias, which is determined by the nature and direction of the study results. To control or minimize this type of bias, the authors of systematic reviews undertake comprehensive searches of medical databases and expand on the findings, often undertaking searches of grey literature (material which is not formally published). This paper attempts to show the consequences (and risk) of generalizing the implications of grey literature in the control of publication bias, as was proposed in a recent systematic work. By repeating the analyses for the same outcome from three different systematic reviews that included both published and grey literature our results showed that confusion between grey literature and publication bias may affect the results of a concrete meta-analysis.

Type
Short communication
Copyright
Copyright © Elsevier SAS 2005

1. Introduction

Systematic literature reviews are useful tools for researchers, policy makers and health care providers. They help to integrate information and provide insight concerning the consistency of scientific findings Reference Mulrow[15]. This synthesis of information is helpful in clinical decision-making, economic evaluations and future research directions Reference Moher, Cook, Eastwood, Olkin, Rennie and Stroup[13]. It is therefore important to use a very strict methodology in systematic reviews. If, however, the studies included in such reviews are flawed, the value of the reviews is open to question Reference Jüni, Altman and Egger[12]. To avoid this pitfall, systematic reviews of randomized control trials (RCTs) should assess the quality of the studies included in terms of four main types of bias: selection, performance, detection and attrition, and also evaluate possible publication bias taking all the studies included into account Reference Clarke and Oxman[1].

Publication bias is determined by the nature and direction of the study results; in other words, studies lacking positive results are usually less likely to be published Reference Clarke and Oxman[1]. Scientists tend to publish their positive results more than their negative findings Reference Dickersin and Rennie[4]. Furthermore, since studies with non-significant results are difficult to publish in English language journals, such studies are likely to be published in journals in other languages: English language bias [6,8].

In consequence, if this type of bias is not addressed when undertaking a systematic review, the final results may be affected by creating an overestimation of the true effect of an intervention. For some authors, this is particularly the case if the review includes only published studies Reference Clarke and Oxman[1], since just because negative studies have not been identified from amongst published studies it does not necessarily mean that such studies do not exist.

To control or minimize this type of bias, the authors of systematic reviews should undertake comprehensive searches of medical databases such as Medline, Embase or CENTRAL (Cochrane Library). The search strategies aim to obtain a trade-off between sensitivity and precision. To expand on the findings from medical databases (electronic searches), these authors often undertake searches of what is called the ‘grey literature’. Grey literature is literature that has not been formally-published and that may include any material ranging from abstract reports to unpublished data Reference Hopewell, McDonald, Clarke and Egger[9], such as personal communications or reports and advertisements from pharmaceutical or technological companies. The registers of ongoing trials available on the Internet are also searched [3] and finally, authors of identified trials may be contacted when possible for additional information about other relevant studies.

In spite of the importance of grey literature as seen above, this paper attempts to show, in the context of psychiatry, the consequences (and risk) of generalizing the implications of this literature in the control of publication bias, as has been supposed by a recent systematic review Reference Hopewell, McDonald, Clarke and Egger[9]. We also attempt to show the need to differentiate between negative and positive data in the grey literature and how this literature, may introduce bias rather than prevent it due to its quality.

To demonstrate how confusion between grey literature and publication bias may affect the results in meta-analysis of schizophrenia, we repeated the analyses for the same outcome from three Cochrane Library systematic reviews that included both published and grey literature. We applied the same statistical methods as were used in these reviews, but we included only formally-published data (according to review references). These three systematic reviews assessed atypical neuroleptic drugs for schizophrenia [5,10,18]. Our purpose was to demonstrate a homogeneous specific comparison rather than an evaluation of quality and an analysis of conclusions.

2. The risk of generalizing the message that grey literature minimizes publication bias

2.1. Outcome measure

In studies with schizophrenic patients, an important issue is the high number of patient dropouts, either as a result of the pathology itself or due to complications of side effects. This pathology modifies the perception of reality in such a way that even the management of these patients during the study could be perceived as a threat (e.g. paranoid schizophrenia). It seems that information from the outcome “leaving the study early” may indicate that schizophrenic patients improve (perception of disease and the need to take medication). This outcome may also be considered as a good indicator of external validity. We interpreted the outcome “leaving the study early for any reason” as meaning that the higher the number of dropouts from the study, the worse the results.

2.2. Examples

In one systematic review that addressed the effectiveness of olanzapine versus typical antipsychotic drugs Reference Duggan, Fenton, Dardennes, El-Dosoky and Indran[5], there was a statistically significant favorable effect using olanzapine compared with using typical antipsychotics of 0.69 (0.51–0.94), P = 0.02; N = 239 subjects for the outcome “leaving the study early—within 3–12months”. In this comparison, the meta-analysis included five studies from the grey literature and only one formally-published study Reference Jakovljevic and Dossenbach[11]. When data from the grey literature (including data for one study provided by the pharmaceutical company) were excluded, the change in number resulted in a non-statistically significant value of 0.69 (0.35–1.37), P = 0.3 for a sample size of 60 subjects from only one study (Fig. 1a). In this example, the inclusion of grey literature did not decrease the effect size found; moreover, analysis including this literature provides even less conservative results. Rather than solve the question for publication bias, therefore, this comparison could be overestimated due to the probable lower quality of this type of literature.

Fig. 1 Overall effect size obtained from the three examples (a) olanzapine (b) quetiapine and (c) risperidone, using grey + formally-published literature (all data) and only formally-published literature. OES: overall effect size. P: P-value. N: number of studies/number of subjects. CI: confidence interval.

Our second example includes a systematic review, which addressed the efficacy of quetiapine vs. classical antipsychotics Reference Srisurapanont, Disayavanish and Taimkaew[18]. The result of the meta-analysis for the outcome “leaving the study early” for any reason in the short term, found that when data from all of the studies were included, the meta-analysis was statistically significance in favor of quetiapine: 0.87 (0.76–0.99), P = 0.04. However, when the data from the grey literature were excluded, the analysis yielded non-significant 0.87 (0.73–1.02), P = 0.09, even though the sample size was as large as 959 subjects (Fig. 1b). As in the previous review, when data from the grey literature were included ostensibly to avoid publication bias, the effect size of the atypical antipsychotic drug did not decrease or balance out. Moreover, it increases the statistical power of this analysis by accentuating the tendency of positive results shown in formally-published studies. The difference between the previous example and this one is that the latter included a sensitivity analysis, which illustrated the difference between all of the data and only the published data. This recommendation by Cook et al. Reference Cook, Guyatt and Ryan[2] was made to ameliorate the problems arising from including studies from both published and grey literature.

Our third example evaluated a review that compared the effectiveness of risperidone versus several typical antipsychotic drugs Reference Hunter, Joy, Kennedy, Gilbody and Song[10]. Data analyses from 23 studies found that the effect size for the outcome “total leaving the study early (any reason)” was 0.69 (0.59–0.80), P < 0.00001, in favor of risperidone. However, excluding the grey literature, the effect size found was 0.67 (0.56–0.82), P < 0.0001, for 15 studies (Fig. 1c). Although both comparisons were statistically significantly in this meta-analysis, the P-value (by Confidence Interval) was less conservative when data from the grey literature were included.

3. Conclusions

These examples illustrate that the use of grey literature does not always lead to less publication bias in meta-analysis of schizophrenia. Moreover, in some cases, inclusion of the grey literature may indeed increase the bias. Therefore, it seems necessary to differentiate between negative and positive data in the grey literature. A comprehensive search for systematic reviews in schizophrenia should pay attention primarily to the former since these data have less chance of being published as formal literature. Positive data from the grey literature should be treated differently from positive published data because there may be two reasons for failure to be published: publication delay, or the low quality of the paper. In the latter case, it is difficult to publish formally through the peer-review process, so many papers appear only in the grey literature. Negative results may also be of low quality, but their status as grey literature is at least questionable. Before grey literature is included in a systematic review of schizophrenia it is advisable to request a draft of the complete paper so that its quality can be adequately assessed. Moreover, in some cases the supplier of data is the industry itself and when such data are only positive, we should ask ourselves whether their inclusion decreases the risk of publication bias in a systematic review or, on the other hand, increases it.

A recent systematic review on this topic Reference Hopewell, McDonald, Clarke and Egger[9] has concluded that including grey literature in meta-analysis in order to minimize the risk of bias may have important implications. The review has also found in general that published studies are of higher quality than grey studies, mainly with respect to allocation concealment and blinding. In our opinion this appears to be a contradiction since the method of allocation concealment used and the implementation of a double-blind methodology in RCTs is generally related to the effect size found [[7,17]; that is inadequate methodological reporting correlates with larger estimates of treatment effects Reference Schulz[16]. The review concludes that even in absence of additional information, “data from grey trials are important, particularly when very few trials of a health care intervention have been published”. We believe this is debatable if our intention is to reduce the bias risk in quantitative data by means of greater control with experimental designs (the study quality).

In systematic reviews or meta-analysis of schizophrenia, we should pay attention to study quality, and to the difference between formally-published literature and grey literature. In the former, the authors commit to the steps which must be passed through and there are formal pathways between authors and readers to clarify any doubts or to confirm the quality of the criteria in the publication of the study. The latter, on the other hand, may be simply an abstract that has not undergone peer review, is very cryptic, and has little detail.

The rules of the CONSORT Statement Reference Moher, Schulz and Altman[14] were established to improve the quality of RCT reports. They are followed by the main biomedical journals as one of the main requirements for accepting trials. These recommendations insist on important aspects of RCT quality, such as the generation of the randomization sequence, the method of allocation concealment used and how sample size was determined. Such aspects are difficult to find in the grey literature and lead to complicated interpretations of study quality. This in turn increases the risk of bias. In addition, in many cases only preliminary results are published in the grey literature, so it seems inappropriate to include such results in a systematic review in an equal way to data from the formally-published literature, especially when both results are positive. It would also seem extremely dangerous to make an evidence-based decision concerning some clinical practice based on a hypothetical sum of grey literature showing positive results.

In summary, studies from grey literature should be included in systematic reviews of mental health only when it is evident that they have not been published due to the nature and direction of the study results. Clearly, the best check and balance when evaluating grey literature is to use the same criteria applied to formally-published RCTs. This should assure adequate safeguards against the application of faulty results in the clinic.

Acknowledgments

We thank Carolyn V. Newey and William Stone for their useful comments and editing assistance. This paper was partially supported by a grant from the Ministerio de Sanidad y Consumo, Instituto de Salud Carlos III (Project no. 01/10004).

References

Clarke, MOxman, ADCochrane Reviewers’ Handbook 4.2.0. The Cochrane Library, Issue 2 Oxford: Update Software. Update quarterly; 2003. [updated March 2003]Google Scholar
Cook, DJGuyatt, GHRyan, GShould unpublished data be included in meta-analysis?. J Am Med Assoc 1993;269:27492753.10.1001/jama.1993.03500210049030CrossRefGoogle Scholar
Current Controlled Trials. MetaRegister of controlled clinical trials. http://www.controlled-trials.com.Google Scholar
Dickersin, KRennie, DRegistering clinical trials. J Am Med Assoc 2003;290:516523.10.1001/jama.290.4.516CrossRefGoogle ScholarPubMed
Duggan, LFenton, MDardennes, RMEl-Dosoky, AIndran, SOlanzapine for schizophrenia (Cochrane Review). The Cochrane Library, Issue 1 Oxford: Update Software; 2004.Google Scholar
Egger, MMeta-analysis bias in location and selection of studies. Br Med J 1998;316:6166.10.1136/bmj.316.7124.61CrossRefGoogle ScholarPubMed
Egger, MJüni, PBartlett, CHolenstein, FSterne, JHow important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol. Assess 2003;7(1):176.Google ScholarPubMed
Heres, SWagenpfeil, SHamann, JKissling, WLeucht, SLanguage bias in neuroscience- Is the tower of Babel located in Germany?. Eur. Psychiatry 2004;19:230232.10.1016/j.eurpsy.2003.09.011CrossRefGoogle Scholar
Hopewell, SMcDonald, SClarke, MEgger, MGrey literature in meta-analysis of randomized trials of health care interventions (Cochrane Methodology Review). The Cochrane Library, Issue 1 Oxford: Update Software; 2003.Google Scholar
Hunter, RHJoy, CBKennedy, EGilbody, SMSong, FRisperidone versus typical antipsychotic medication for schizophrenia (Cochrane Review). The Cochrane Library, Issue 1 Oxford: Update Software; 2004.Google Scholar
Jakovljevic, MDossenbach, MRKOlanzapine versus fluphenazine in the acute (6-week) treatment of schizophrenia. Psychiatric Danubina 1999;11:310.Google Scholar
Jüni, PAltman, DGEgger, MAssessing the quality of controlled clinical trials. Br Med J 2001;323:4246.10.1136/bmj.323.7303.42CrossRefGoogle ScholarPubMed
Moher, DCook, DJEastwood, SOlkin, IRennie, DStroup, DFet al.Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Lancet 1999;354:18961900.10.1016/S0140-6736(99)04149-5CrossRefGoogle ScholarPubMed
Moher, DSchulz, KFAltman, DGfor the CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001;357:11911194.10.1016/S0140-6736(00)04337-3CrossRefGoogle Scholar
Mulrow, CDSystematic reviews: rationale for systematic reviews. Br Med J 1994;309:597599.10.1136/bmj.309.6954.597CrossRefGoogle ScholarPubMed
Schulz, KFRandomised trials, human nature, and reporting guidelines. Lancet 1996;348: 596598.10.1016/S0140-6736(96)01201-9CrossRefGoogle ScholarPubMed
Schulz, KFChalmers, IHayes, RJAltman, DGDimensions of methodological quality associated with estimates of treatment effects in controlled trials. J. Am. Med. Assoc. 1995;273: 408412.10.1001/jama.1995.03520290060030CrossRefGoogle ScholarPubMed
Srisurapanont, MDisayavanish, CTaimkaew, KQuetiapine for schizophrenia (Cochrane Review). The Cochrane Library, Issue 1 Oxford: Update Software; 2004.Google Scholar
Submit a response

Comments

No Comments have been published for this article.

Full text views

Full text views reflects PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views.

Total number of HTML views: 10
Total number of PDF views: 6 *
View data table for this chart

* Views captured on Cambridge Core between 16th April 2020 - 23rd January 2021. This data will be updated every 24 hours.

Access
Hostname: page-component-76cb886bbf-tvlwp Total loading time: 1.785 Render date: 2021-01-23T12:45:11.661Z Query parameters: { "hasAccess": "1", "openAccess": "0", "isLogged": "0", "lang": "en" } Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": false, "newCiteModal": false }

Send article to Kindle

To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Is grey literature essential for a better control of publication bias in psychiatry? An example from three meta-analyses of schizophrenia
Available formats
×

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Is grey literature essential for a better control of publication bias in psychiatry? An example from three meta-analyses of schizophrenia
Available formats
×

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Is grey literature essential for a better control of publication bias in psychiatry? An example from three meta-analyses of schizophrenia
Available formats
×
×

Reply to: Submit a response


Your details


Conflicting interests

Do you have any conflicting interests? *