Pre-print servers: background and evolution

Preprint servers are sites where authors can post their scientific papers prior to submission, review, or acceptance in a traditional journal (Fig. 1). They existed prior to the pandemic, but their use was limited, and they were not vehicles for widespread dissemination of clinical or public health studies. The first pre-print server, arXiv, was created in 1991 and covered non-medical sciences. Servers dedicated to medical and health sciences (such as medRxiv) and life sciences (such as bioRxiv) were established much later but did not rocket to popularity or serve as major information sources until the pandemic began. The most commonly used pre-print servers are not-for-profit and provide varying levels of basic screening prior to approving manuscripts (which must be original research, not case reports, editorials, or narrative reviews). Authors may submit revised versions of the same manuscript at any time, and these are tracked by the system, providing a historical record of the major changes. Some pre-print servers have affiliations with journals, particularly those that signed on to the collaborative mentioned above, facilitating direct transfer of files and saving authors time during the submission process.

Figure 1. Steps in the academic publishing process.

The global pandemic: a catalyst for speeding and transforming data dissemination

As the year 2020 dawned, the established mechanisms by which research findings are made available were too slow to keep pace with the needs of the healthcare and scientific communities. Even for studies that go on to be accepted, the editorial process for publication in a peer reviewed journal, which includes multiple reviews and revisions, easily takes many months, and sometimes more than a year. Clearly, with a deadly and unknown disease ravaging the globe, and new discoveries being made every day, business as usual would have been inadequate and inappropriate. The pre-print servers offered a solution to these problems.

For the first time, pre-print servers became important sources of medical information, with the papers therein amplified in traditional and non-traditional news outlets and even included in deliberations about important public health decisions, like school policy, mask mandates, and vaccine recommendations. Studies posted on pre-print servers impacted citation and alt-metric scores—two measures of research impact—and some had substantial impacts on clinical care decisions and policy without high-quality—or occasionally even verified—data to back up their claims. ^{Reference Fraser, Momeni, Mayr and Peters3,Reference Conroy4}

Pre-prints: the good, the bad, the ugly

Pre-prints: empirical evidence of impact

These concerns are not just theoretical but supported by empirical evidence. Likely driven in part by anchoring bias, despite these well-documented limitations of different study designs and analyses, by the time contradictory or higher-quality evidence became available, the first report has already made its lasting mark. For example, a study by one of the authors (WBE) measured the speed and scope of COVID-19 prescribing practices in the Veterans Health System and found that practice patterns changed rapidly after data release via pre-print servers and press releases, with limited additional change after peer-reviewed publication. ¹¹ Effects were strongest early in the pandemic, when there were few treatment options, and waned as time went on. Studies on the efficacy of the bivalent COVID-19 vaccine in human subjects have been presented at regulatory meetings before peer review, allowing for expedited authorization and guideline development, potentially benefiting society by reducing delays. ^{Reference La, Fillmore and Do12,Reference Chalkias, Harper and Vrbicky13} On the other hand, even after a pre-print purportedly demonstrating efficacy was withdrawn from the server due to serious ethical infractions, ^{Reference Wang, Hueda-Zavaleta, Cáceres-DelAguila, Muro-Rojo, De La Cruz-Escurra and Benítes-Zapata14} ivermectin has continued to be prescribed despite a preponderance of evidence demonstrating lack of efficacy. ^{Reference Reardon15,Reference Lind, Lovegrove, Geller, Uyeki, Datta and Budnitz16} Other examples of practice changes driven by early studies with major limitations include “gaiter-gate”. ^{Reference Becker, Seelye, Chua, Echevarria, Conti and Prescott17,Reference Chiu18} Despite rapidly available contradictory evidence, the damage was done. Gaiters were widely banned in schools based on an early report that they were inferior to other cloth masks and subsequent dissemination via traditional and social media sources. ^{Reference Fischer, Fischer, Grass, Henrion, Warren and Westman19,Reference Kielar20} In another example, a research group using publicly available CDC data published a pre-print in May 2022 that calculated that COVID-19 was the 4^th highest cause of death in children under the age of 19. ^{Reference Parker-Pope21} This analysis was shared in both the FDA’s VRBPAC meeting and the CDC’s ACIP meeting to inform discussion about COVID-19 vaccination policy for children. Findings were also widely shared in traditional and social media outlets. A revised version of the pre-print, ^{Reference Flaxman, Whittaker and Semenova22} however, listed COVID-19 as the 7^th highest cause of death, and in the final publication, COVID-19 was calculated to be the 8^th highest cause of death. ²³ During the revision process, the estimated crude mortality rate of COVID-19 for children changed quite dramatically, from 7.2 down to 1.0 per 100,000 from the first version of the pre-print to the final, peer-reviewed published version of the manuscript.

Evolving pre-prints to meet current needs and conditions

Recognizing the limitations of pre-prints, their benefits, and their real-world impacts raises important questions about how the use of pre-print servers for disseminating medical evidence should evolve as we move into the next phase of living with COVID-19 (Table 1). ^{Reference Flaxman, Whittaker and Semenova24} First, given the aforementioned data demonstrating that one of the biggest impacts of peer-review is to change framing and reduce spin, a pre-print policy change that could be trialed is presentation of methods, results, and limitations sections only without background and discussion sections, which are fundamentally more prone to personal opinion and political viewpoints. Another possibility is for pre-print servers to be re-organized, such that limitations are a required element that are presented before the results section. This would ensure that limitations are highlighted so that the public can view and comment on them; reading limitations first might also change the reader’s perceptions about the implications of the results, partially (if not completely) addressing anchoring bias tendencies. Additional requirements for quality checks and release of underlying methodology (e.g., by requiring code to be shared with the release, or publication of models so that others can review) would also be options for improving pre-print transparency and evaluations. Development and application of advanced technology to identify falsified images, calculation errors, and plagiarism are additional considerations that should be considered, supported, and evaluated to improve evidence quality.

Table 1. Pre-prints: benefits and downsides relative to traditional academic publishing

Evolving existing academic publishing and data sharing

The influence of pre-prints has not occurred in a vacuum. They are one piece of the puzzle and changes at many levels of the system are needed to improve data dissemination. Before pre-prints are widely shared by influencers, covered by the media, and communicated to the public as “truth,” uncertainty about the study’s eventual conclusions and implications of the work needs to be acknowledged by those who are responsible for sharing and spreading the information.

Although pre-prints in their current iteration may not be the ultimate solution, innovation in medical publishing and dissemination of research findings is badly needed. Academic publishing is strongly biased toward publishing papers—and questioning findings less—if authored by those who are already “famous.” ²⁵ Peer review is slow, biased, and often does not lead to major changes even when they are needed, ^{Reference Brainard26} as occurred with the Surgisphere debacle, in which the underlying data supporting the harms of hydroxychloroquine for COVID-19 treatment could not be verified. ^{Reference Janda, Khetpal, Shi, Ross and Wallach27} For topics of critical public health importance, rapid review pathways should be adopted to ensure that data from high-quality randomized controlled trials are quickly available to inform clinical care decisions. Peer-reviewed manuscripts in traditional journals should be readily available and accessible without a subscription, and access needs to be expanded; Plan S in Europe ^{Reference Offord28} and the White House Office of Science and Technology policy on free, immediate, and equitable access are important policy changes aimed to address these barriers. ^{Reference Schliltz29} Open access should be the standard, and The Journal of the American Medical Association took big steps in this direction in December 2022. ³⁰ The National Library of Medicine could require indexed journals to, at a minimum, make methods, results, and limitations openly accessible to the public and set quality standards to encourage expansion of open-access models.

The financial incentives of academic publishing also inherently create challenges. Academic journals rely on the donated time of unpaid reviewers, which may limit review quality. Journals requesting peer-review are asking for time and expertise—things that outside of academia are always compensated. The practice of donated reviews may be defensible for society journals that do not have publication fees and therefore providing an expert review free-of-charge is a quid pro quo. However, given the rise of for-profit journals and major publishing enterprises, some of which boast profit margins higher than those of Apple or Google, ^{Reference Bibbins-Domingo, Shields and Ayanian31} due in part to the free labor and expertise provided by reviewers—the compensation structure, or lack thereof, needs reevaluation. However, despite all of these caveats and limitations, recent data suggest that the speed and quality of peer review may actually have increased during the pandemic. ^{Reference Buranyi32,Reference Perlis, Kendall-Taylor and Hart33}

Next steps

What is the best path forward? Evidence of quality and our willingness to act upon findings need to be balanced (Table 2). Data from pivotal clinical trials should be reviewed and released rapidly, so that clinicians and policy makers can translate evidence into care immediately; recently released data on the harms associated with corticosteroids for inpatients without severe COVID-19 is an example that should immediately change practice. ³⁴ For other studies, speed and content quality could be balanced by rapid posting of accepted, but not publication proofed, articles.

Table 2. Framework for considering data release and sharing

Mainstream and social media outlets should wait for observational and laboratory-based research to undergo a vetting process. Denise-Marie Ordway wrote in The Journalist’s Resource in April 2020 that there are 6 things journalists must know before covering biomedical research pre-prints about COVID-19: ³⁵ (1) pre-prints can be dangerous if doctors change their practice based on the results; (2) pre-prints are not peer reviewed, leaving them without opportunity for other experts to catch errors, argue with the authors’ interpretation of findings, or request additional data or analyses; (3) best practice dictates that reporters should be explicit that findings reported in pre-prints are preliminary and, ideally, the opinion of an expert should be included to counter inappropriate conclusions; (4) pre-prints are best covered by experienced science journalists; (5) journalists should check with experts to determine whether findings from a pre-print are even worthy of coverage; (6) pre-prints are sometimes withdrawn, particularly after feedback from the scientific community alerts authors to flaws in their methodology or conclusions. ³⁴