Skip to main content Accessibility help
×
Home

On the potential distortions of highly cited papers in emerging research fields: A critical appraisal

  • Edoardo G. Ostinelli (a1), Orsola Gambini (a1) and Armando D'Agostino (a1)

Abstract

Citation-based metrics are increasingly used as a proxy to define representative, considerable, or significant papers. We challenge this belief by taking into account factors that may play a role in providing citations to a manuscript and whether/how those highly cited studies could shape a scientific field. A different approach to summarisation of relevant core publications within a topic is proposed.

Copyright

References

Hide All
Berer, K., Mues, M., Koutrolos, M., Rasbi, Z. Al, Boziki, M., Johner, C., Wekerle, H. & Krishnamoorthy, G. (2011) Commensal microbiota and myelin autoantigen cooperate to trigger autoimmune demyelination. Nature 479(7374):538–41. Available at: https://doi.org/10.1038/nature10554.
Bramer, W. M., Giustini, D. & Kramer, B. M. (2016) Comparing the coverage, recall and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: A prospective study. Systematic Reviews 5:39. Available at: https://doi.org/10.1186/s13643-016-0215-7.
Catalogue of Bias Collaboration, Aronson, J. K., Bankhead, C. & Nunan, D. (2017a) Hot stuff bias. In: Catalogue of bias. University of Oxford. Available at: https://catalogofbias.org/biases/hot-stuff-bias/.
Catalogue of Bias Collaboration, Plüddemann, A., Banerjee, A. & O'Sullivan, J. (2017b) Positive results bias. In: Catalogue of bias. University of Oxford. Available at: https://www.catalogueofbiases.org/biases/positive-results-bias.
Catalogue of Bias Collaboration, Spencer, E. A., Brassey, J. & Heneghan, C. (2017c) One-sided reference bias. In: Catalogue of bias. University of Oxford. Available at: https://www.catalogofbias.org/biases/one-sided-reference-bias.
Cheek, J., Garnham, B. & Quan, J. (2006) What's in a number? Issues in providing evidence of impact and quality of research(ers). Qualitative Health Research 16(3):423–35. Available at: https://doi.org/10.1177/1049732305285701.
Dickersin, K., Scherer, R. & Lefebvre, C. (1994) Identifying relevant studies for systematic reviews. British Medical Journal 309(6964):1286–91. Available at: https://doi.org/10.1136/bmj.309.6964.1286.
Greenberg, S. A. (2009) How citation distortions create unfounded authority: Analysis of a citation network. British Journal of Medicine 339:b2680. Available at: https://doi.org/10.1136/bmj.b2680.
Hafeez, D. M., Jalal, S. & Khosa, F. (2019) Bibliometric analysis of manuscript characteristics that influence citations: A comparison of six major psychiatry journals. Journal of Psychiatric Research 108:9094. Available at: https://doi.org/10.1016/j.jpsychires.2018.07.010.
Ioannidis, J. P. A. (2005a) Contradicted and initially stronger effects in highly cited clinical research. JAMA 294(2):218–28.
Ioannidis, J. P. A. (2005b) Why most published research findings are false. PLoS Medicine 2(8):e124. Available at: https://doi.org/10.1371/journal.pmed.0020124.
Ioannidis, J. P. A., Boyack, K. W., Small, H., Sorensen, A. A. & Klavans, R. (2014) Bibliometrics: Is your most cited work your best? Nature 514:561–62. Available at: https://doi.org/10.1038/514561a.
Ioannidis, J. P. A. & Khoury, M. J. (2014) Assessing value in biomedical research: The PQRST of appraisal and reward. JAMA 312(5):483–84. Available at: https://doi.org/10.1001/jama.2014.6932.
Ioannidis, J. P. A. & Panagiotou, O. A. (2011) Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses. JAMA 305(21):2200–10. Available at: https://doi.org/10.1001/jama.2011.713.
Kulkarni, A. V., Aziz, B., Shams, I. & Busse, J. W. (2009) Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. JAMA 302(10):1092–96. Available at: https://doi.org/10.1001/jama.2009.1307.
Lefebvre, C., Manheimer, E. & Glanville, J. (2011) Searching for studies. In: Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011), ed. Higgins, J. P. T. & Green, S.. The Cochrane Collaboration. Available at: http://www.handbook.cochrane.org.
Mackinnon, S., Drozdowska, B. A., Hamilton, M., Noel-Storr, A. H., McShane, R. & Quinn, T. (2018) Are methodological quality and completeness of reporting associated with citation-based measures of publication impact? A secondary analysis of a systematic review of dementia biomarker studies. BMJ Open 8:e020331. Available at: https://doi.org/10.1136/bmjopen-2017-020331.
Rathbone, J., Carter, M., Hoffmann, T. & Glasziou, P. (2016) A comparison of the performance of seven key bibliographic databases in identifying all relevant systematic reviews of interventions for hypertension. Systematic Reviews 5:27. Available at: https://doi.org/10.1186/s13643-016-0197-5.
Seglen, P. O. (1998) Citation rates and journal impact factors are not suitable for evaluation of research. Acta Orthopaedica Scandinavica 69(3):224–29. Available at: https://doi.org/10.3109/17453679809000920.
Tajika, A., Ogawa, Y., Takeshima, N., Hayasaka, Y. & Furukawa, T. A. (2015) Replication and contradiction of highly cited research papers in psychiatry: 10-Year follow-up. British Journal of Psychiatry 207:357–62. Available at: https://doi.org/10.1192/bjp.bp.113.143701.
Tatsioni, A., Bonitsis, N. G. & Ioannidis, J. P. A. (2007) Persistence of contradicted claims in the literature. JAMA 298(21):2517–26. Available at: https://doi.org/10.1001/jama.298.21.2517.
Wieland, S. & Dickersin, K. (2005) Selective exposure reporting and Medline indexing limited the search sensitivity for observational studies of the adverse effects of oral contraceptives. Journal of Clinical Epidemiology 58(6):560–67. Available at: https://doi.org/10.1016/j.jclinepi.2004.11.018.

On the potential distortions of highly cited papers in emerging research fields: A critical appraisal

  • Edoardo G. Ostinelli (a1), Orsola Gambini (a1) and Armando D'Agostino (a1)

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed.