THEORETICAL ASSUMPTIONS

I begin by describing the framework of assumptions about the review process on which I base my analysis. I assume that there exists a population of potentially publishable papers, some of which will be submitted to a journal; I also assume that submitted papers are representative of the overall population. ^{Footnote 2} The journal’s editor seeks to publish papers that are in the top ${p^ \star }$ percentile of papers in the population in terms of quality. When editors receive a paper, I assume that they solicit three blind reviews; I later relax this assumption to allow editors to desk-reject papers before review. I further assume that editors assign papers to reviewers at random, conditional on expertise, and that any refusals to review are unrelated to paper quality. This assumption rules out the possibility that editors selectively choose reviewers in anticipation of the review that they believe they will receive or that reviewers self-select out of bad (or good) reviews. ^{Footnote 3}

Each reviewer $i \in \{ 1,2,3\}$ and the editor i = 4 forms an opinion about paper j’s overall quality, ${p_{ij}} \in [0,1]$ , where p _ij corresponds to the proportional rank of the paper’s holistic quality relative to the population of papers. For example, ${p_{ij}} = 0.8$ means that reviewer i believes that paper j is better than 80% of the other papers in the population. If papers are randomly assigned to reviewers (conditional on expertise), then approximately p proportion of the papers assigned to a reviewer will have quality less than or equal to p for every value of $p \in [0,1]$ . As a result, every reviewer’s marginal distribution of reviews ${f_i}(p)$ should be uniform. Reviewers have partially dissimilar preferences, limited time to review a paper, and the possibility of making errors; they also cannot influence one another’s opinion before forming their judgment. For all of these reasons, I assume that reviewers’ judgments of a paper are imperfectly associated with one another, which is consistent with the findings of a long empirical literature (Bornmann, Mutz, and Daniel Reference Bornmann, Mutz and Daniel2010; Goodman et al. Reference Goodman, Berlin, Fletcher and Fletcher1994; Mahoney Reference Mahoney1977; Mayo et al. Reference Mayo, Brophy, Goldberg, Klein, Miller, Platt and Ritchie2006; Nylenna, Riis, and Karlsson Reference Nylenna, Riis and Karlsson1994; Schroter et al. Reference Schroter, Black, Evans, Godlee, Osorio and Smith2008). Functionally, I presume that the three reviewers’ opinions and the editor’s are drawn from a normal copula with correlation $\rho \in [0,1]$ . I intend that higher values of $\rho$ model the behavior of reviewers for journals with narrower topical and methodological coverage (e.g., Legislative Studies Quarterly and Political Analysis), whereas lower values of $\rho$ model the behavior of reviewers for general-interest journals (e.g., American Political Science Review [APSR]). In practice, editors could exert some control over $\rho$ by choosing more or less like-minded reviewers, but (consistent with previous assumptions) $\rho$ is fixed and not chosen by the editor in this model.

Each reviewer i submits a vote about paper j to the editor, ${v_{ij}} \in \{ A,R\}$ , based on paper j’s quality. I assume that reviewers recommend the best papers to the editors for publication; thus, the reviewers compare ${p_{ij}}$ to an internal threshold for quality ${p^'}$ and submit a vote of A if ${p_{ij}} \ge {p^'}$ and a vote of R otherwise. Given the uniform distribution of quality, this implies that the probability that a reviewer returns a positive review (of ${v_{ij}} = A$ ) is equal to $(1 - {p^'})$ . One particularly interesting threshold to investigate is ${p^'} = {p^ \star }$ , where the reviewers set their probability of recommending that a paper be accepted equal to the journal’s target acceptance rate. ^{Footnote 4} Reviewers also submit a qualitative written report to the editor that contains ${p_{ij}}$ , which allows a more finely grained evaluation of papers. ^{Footnote 5}

I assume that reviewers sincerely report their ${v_{ij}}$ to editors. Editors then use their own opinion about paper j’s quality ( ${p_{4j}}$ ), their holistic judgment about the paper ( ${v_{4j}} = A$ if ${p_{4j}} \ge {p^'}$ , and = R otherwise), the reviewers’ qualitative reports ( ${p_{ij}}$ ), and the reviewers’ votes to decide whether to accept or reject the paper. I consider four possible editorial regimes for converting reviews into decisions, as follows:

• • unanimity approval voting by the reviewers, excluding the editor

• • simple majority voting by the reviewers, excluding the editor

• • majority voting with the editor’s vote included (i.e., to be accepted, a paper must achieve support from all three reviewers or two reviewers and the editor)

• • unilateral editor decision making based on the average report ${\bar p_j} = {1 \over 4}\mathop \sum \limits_{i = 1}^4 {p_{ij}}$ , with the paper accepted if ${\bar p_j} \ge {p^'}$ and reviewers’ votes ignored

The final regime acknowledges that editors try to follow the advice of the reviewers whose participation they solicit but may choose not to follow the reviewers’ up or down recommendation. This regime is analogous to a system under which an editor reads reviewers’ written reports to collect information about the paper’s quality that will influence his/her decision but either does not request or simply ignores the reviewers’ actual vote to accept or reject.

The model I propose is substantially more complex than another model recently proposed by Somerville (Reference Somerville2016). In Somerville’s model, the quality of a journal article is binary—good (G) or bad (B)—and the review process is abstracted into a single probability of accepting an article based on its quality ( $\Pr (A|G)$ and $\Pr (A|B)$ ). The goal of Somerville’s study was to use Bayes’s rule to calculate the probability of an article being good conditional on its being accepted ( $\Pr (G|A)$ ). By comparison, my model explicitly includes multiple reviewers with competing (but correlated) opinions of continuous paper quality, editors with decision-making authority, and institutional rules that can be systematically changed and studied. I compare our results in the summary of my findings below.

ACCEPTANCE TARGETS AND REVIEWER STANDARDS

I begin by investigating the simple relationship between each editorial system (i.e., unanimity approval voting, majority approval voting, majority approval voting with editor participation, and unilateral editor decision making based on reviewer reports) and the journal’s final acceptance rate. For every value of the degree of correlation in reviewer reports $\rho \in [0.02,0.98]$ in increments of 0.02, I simulate 2,000 papers from the population distribution and three reviews for each paper, as well as the editor’s personal opinion (for a total of four reviews). I then apply the specified decision rule for acceptance to each paper and determine the acceptance rate. I plot the overall journal acceptance rate as a function of $\rho$ and examine the relationship for ${p^'} = 0.90$ , a reviewer/editor acceptance rate of 10%. All simulations are conducted using R 3.2.5 (R Core Team 2015) with the copula package (Kojadinovic and Yan Reference Kojadinovic and Yan2010).

The simulation results are shown in figure 1. As the figure indicates, the probability of a manuscript being accepted is always considerably less than any individual reviewer’s probability of submitting a positive review unless $\rho \approx 1$ . The systems vary in the disparity between the individual acceptance threshold ${p^'}$ and the journal’s overall acceptance rate, but all of them undershoot the target. For a journal to accept 10% of its submissions, reviewers and editors must recommend papers that they perceive to be considerably below the top 10%.

Figure 1 Simulated Outcomes of Peer Review under Various Peer Review Systems

Notes: Points indicate the proportion of 2,000 simulated manuscripts under the peer review system indicated; lines are predictions from a local linear regression of the data using loess in R. Reviewer acceptance thresholds ${p^'} = 0.90$ (i.e., a reviewer recommends the top 10% of papers for acceptance) for all systems.

THE EFFECT OF THE PEER REVIEW SYSTEM ON THE QUALITY OF THE PUBLISHED LITERATURE

Will peer review accept the papers that the discipline views as the best despite heterogeneity in reviewer opinions? To what extent will quality and chance determine the outcomes of peer review? To answer these questions, I conduct another simulation similar to the previous one but with a much larger population of 50,000 papers and 500 readers. I assume that readers’ opinions are correlated at $\rho = 0.5$ , consistent with a flagship journal; this is a greater degree of reviewer correlation than empirical studies typically find (Bornmann, Mutz, and Daniel Reference Bornmann, Mutz and Daniel2010). The first three simulated readers are selected as reviewers and the fourth as the editor; their opinions serve as the basis for editorial decisions in each of the four systems examined. I choose an acceptance threshold ${p^'}$ based on initial simulations to produce an overall journal acceptance rate of $\approx 10\%$ . ^{Footnote 6} I then compute the average reader value of $p$ for all 500 readers for the papers that are accepted for publication according to the system and plot the distribution of these average values for all 50,000 papers.

I ran this simulation twice: once for every review system as previously described and again under a system in which the editor desk-rejects a certain proportion of papers before submitting them for review. I simulate the process of desk rejection by having the editor refuse to publish any paper for which ${p_{4j}} < 0.5$ ; that is, the editor desk-rejects any paper that he or she believes is worse than the median paper in the population. Figure 2 presents kernel density estimates for the results with and without desk rejection.

Figure 2 Discipline-Wide Evaluation of Papers Published under Various Peer Review Systems, Reader and Reviewer Opinion Correlation ρ = 0.5

Notes: Plots indicate kernel density estimates (using density in R) of 500 simulated readers’ average evaluation (p) for the subset of 50,000 simulated papers that were accepted under the peer review system indicated in the legend. Reviewer acceptance thresholds ${p^'}$ were chosen to set acceptance rates $\approx 10\%$ . The acceptance rate without desk rejection was 10.58% for the unilateral-editor system, 10.01% under unanimity voting, 10.05% under majority rule including the editor, and 11.1% under majority rule excluding the editor. The acceptance rate with desk rejection was 9.64% for the unilateral-editor system, 9.96% under unanimity voting, 12.5% under majority rule including the editor, and 10.5% under majority rule excluding the editor.

Figure 2 indicates that all of the systems produce distributions of published papers that are centered on a mean reader evaluation near 0.8. In every peer review system, a majority of papers are perceived by readers as not being in the top 10% of quality despite the journal’s acceptance rate of 10%. Furthermore, a substantial proportion of the published papers have surprisingly low mean reader evaluations under every system. For example, 11.7% of papers published under the majority-voting system without desk rejection have reader evaluations of less than 0.65. An average reader believes that such a paper is worse than 35% of other papers in the population of papers, many of which are not published by the journal. This result is surprisingly consistent with what political scientists actually report about the APSR, a highly selective journal with a heterogeneous readership: although it is the best-known journal among political scientists by a considerable margin, it is ranked only 17th in quality (Garand and Giles Reference Garand and Giles2003). This result also complements the earlier findings of Somerville (2016, 35), who concluded that “if the rate of accepting bad papers is 10%, then a journal that has space for 10% of submissions may not gain much additional quality from the review process.” The peer review systems that I study all improve on the baseline expectation of quality without review (i.e., a mean evaluation near 0.5), but they do not serve as a perfect filter.

There are meaningful differences among the reviewing systems: only 5.6% of papers published under the unilateral editor decision system without desk rejection have reader evaluations of less than 0.65. If editors desk-reject 50% of papers under this system, then the proportion decreases to 1.4%; this is the best-performing system in the simulation on this criterion. ^{Footnote 7} These better-performing systems are analogous to those in which reviewers provide a qualitative written evaluation of the paper’s quality to the editor but no up or down vote to accept the paper (or when that vote is ignored by the editor).

It is important that the simulated peer review systems tend to accept papers that are better (on average) than the rejected papers, which is consistent with the empirical evidence of Lee et al. (Reference Lee, Schotland, Bacchetti and Bero2002) that journal selectivity is associated with higher average methodological quality of publications. However, luck still plays a strong role in determining which papers are published under any system. Both of these findings are shown in figure 3, which plots an average reader’s evaluation of a simulated paper against its loess-predicted probability of acceptance. In all systems, the highest-quality papers are the most likely to be published; however, a paper that an average reader evaluates as near the 80th percentile of quality (or 85th percentile when desk rejection is used) has a chance of being accepted similar to a coin flip.

Figure 3 The Role of Chance in Publication under Various Peer Review Systems, Reader and Reviewer Opinion Correlation ρ = 0.5

Notes: Plots indicate zeroth-degree local regression estimates (using loess in R) of the empirical probability of acceptance for 50,000 simulated papers under the peer review system indicated in the legend as a function of 500 simulated readers’ average evaluation ( $p$ ). Reviewer acceptance thresholds ${p^'}$ were chosen to set acceptance rates $\approx 10\%$ . The acceptance rate without desk rejection was 10.58% for the unilateral-editor system, 10.01% under unanimity voting, 10.05% under majority rule including the editor, and 11.1% under majority rule excluding the editor. The acceptance rate with desk rejection was 9.64% for the unilateral-editor system, 9.96% under unanimity voting, 12.5% under majority rule including the editor, and 10.5% under majority rule excluding the editor.

THE STRUCTURE OF PREFERENCES AND ITS EFFECT ON PEER-REVIEWED PUBLICATION QUALITY

The structure of preferences in the underlying population of a journal’s readership (and reviewer pool) is powerfully associated with how the quality of publications that survive the peer review system is perceived in the simulations. This structure incorporates the overall degree to which opinion is correlated in the population of a journal’s readers and reviewers; however, it also includes the degree to which scientists in a discipline are organized into subfields within which opinions about scientific importance and merit are comparatively more homogeneous. I find that journals with a more homogeneous readership—or with disparate but internally homogeneous subfields—tend to publish more consistently high-quality papers (as defined by the judgment of its readers) than journals with a heterogeneous readership.

To demonstrate this point, I repeat the simulation of 50,000 papers (using unilateral editor decision making) under three conditions: (1) reader and reviewer opinions correlated at 0.5 to represent a flagship journal in a heterogeneous field (e.g., APSR); (2) reader and reviewer opinions correlated at 0.75 to represent a journal in a more homogeneous field (e.g., Political Analysis) ^{Footnote 8} ; and (3) readers and reviewers organized into two equally sized subfields of 250 people each, within which opinions correlated at 0.9 but between which opinions correlated at 0.1 (for an average correlation of 0.5). ^{Footnote 9} When subfields exist, reviewers are chosen so that two reviewers are from one subfield and the final reviewer and editor are from the other. These subfields may represent different topical specialties or different methodological approaches within a discipline, such as qualitative area specialists and quantitative large-N comparativists who both read a comparative politics journal.

As the results in figure 4 demonstrate, the organization of scientists by subfield has a dramatic impact on the perceived quality of publications in the journal. Specifically, the simulation with two highly correlated but disparate subfields produces few papers with an overall quality of less than 0.8; the average quality is 0.85 (without desk rejection) or 0.90 (with desk rejection). By comparison, the subfield-free simulation with low correlation (0.5) indicates that many more low-quality papers are published under this condition. The subfield-free simulation with high correlation (0.75) produced papers with high average quality (0.88 without desk rejection, 0.92 with desk rejection) but still allows a substantial number of lower-quality papers to be published.

Figure 4 Average Discipline-Wide Evaluation of Papers Published under a Unilateral-Editor Approval System Informed by Submitted Reviewer Reports with Varying Structure of Opinion

Notes: Plots indicate kernel density estimates (using density in R) of 500 simulated readers’ average evaluation ( $p$ ) for the subset of 50,000 simulated papers that were accepted under the unilateral-editor review system for the structure of reader and reviewer opinion correlation indicated in the legend. Reviewer acceptance thresholds ${p^'}$ were chosen to set acceptance rates $\approx 10\%$ . The acceptance rate for all readers correlated at 0.5 was 10.58% without desk rejection and 9.65% with desk rejection. The acceptance rate for all readers correlated at 0.75 was 10.50% without desk rejection and 10.23% with desk rejection. The acceptance rate for the two-subfield discipline was 10.16% without desk rejection and 9.46% with desk rejection.