One of the most challenging conundrums editors facing is how to convince rejected authors to review the manuscripts of other authors. Because most manuscripts are ultimately rejected from top academic journals, it is of utmost importance that rejected authors receive reviews (ideally on time), which help them to improve their research and increase their publication chances elsewhere. As editors decide on a manuscript’s suitability for publication, given the journal’s scope and audience in mind, they strongly rely on the evaluation of the more specialized reviewers (field-specific) who assess the quality of a manuscript. Accordingly, a well-functioning review system is central for the success of top academic journals and ensures the trustworthiness and quality of published research. Thus, in this issue’s “Notes from the Editors,” we want to discuss some features of reviewer recommendations we receive and how they influence our decisions. We find this exercise particularly enlightening as it is important that we remain transparent, given our high rejection rate and reliance on the very community that reviews for the APSR.
When informing authors about the decision on their manuscript in the peer review process, editors must find a balance between transparency and anonymity with respect to the amount of information that authors and reviewers receive. On the one hand, transparent decisions aim to enhance trust and understanding among (rejected) authors and the reviewers involved to reduce risks of superficiality and bias in the editorial decision-making process. On the other hand, anonymity increases the likelihood that the manuscripts of authors are evaluated without bias against their personal background and that reviewers respond freely and openly without the fear of retaliation. According to our view, both are equally important features in ensuring the quality of the reviews and, ultimately, editorial decisions.
Nevertheless, some (rejected) authors raise concerns and sometimes challenge decisions or qualifications of reviewers. Except for “false” qualifications, which are difficult to identify in most political science subfields, we receive challenges of authors who often perceive the comments to be more favorable. Accordingly, editors are assumed to be particularly critical by overruling more positive recommendations. Although this may (unfortunately) happen at times, our own experience is that editors tend to have a harder time finding reasons not to reject a manuscript rather than the other way around. To quantify these different views, we evaluate reviewer recommendations using data on 21,238 reviewer recommendations from July 2007 to September 2019. In addition, we want to highlight that editors are able to receive information regarding the perceived feasibility of addressing concerns raised by the reviewers that authors do not have access to. Naturally, this may result in perceptions of unfair and biased decisions. Yet, by shedding some light on the aggregate data, we hope to increase transparency on the decisions we make, but also wish to emphasize the difficulties editors face to keep authors and reviewers satisfied.
The APSR provides a double-blind peer-review process where neither authors nor reviewers should be aware of the identity of the other. Once enough reviews are collected, authors and reviewers are informed about the decision; in fact, both the authors and reviewers receive the editorial outcome, including the anonymous comments from all reviewers. Providing authors and reviewers with detailed feedback not only aims to improve research and guide revisions of the manuscript for future submission but also presents context for the editors’ decision. However, when making the decision, editors may sometimes have additional information unavailable to the authors and other reviewers.
Behind the editorial curtain, editors receive a general recommendation term from the reviewers as well as a rating of five central characteristics of a manuscript: importance of subject matter, appropriateness for the APSR, quality of research, quality of writing, and advance over previous work. Reviewers also have the opportunity to provide private comments to the editor that are withheld from the authors and other reviewers. It provides a chance to both praise and raise concerns which the reviewer—for various reasons—feels he or she does not want to share with others. A large amount of editors’ time is thus spent on aggregating this information to come to a fair and justified decision.
Given the additional information provided to the editors, are they more critical toward submissions than reviewers? During the last editorial term, 2018–19, the APSR granted about 6% of submissions the chance to revise and resubmit. With a similarly low final acceptance rate of about 4–5%, the process is naturally criticized by rejected authors who perceive editors as being overly critical. However, in the vast majority of cases, reviewers are at least equally critical toward the submissions. In the first round of review, after one-third of manuscripts have already been desk-rejected, only 4% of reviews recommend publication and only 15% suggest minor revisions. Put differently, about 82% of reports either have major concerns with the manuscript or recommend rejection.
As shown in Figure 1, this pattern holds across subfields, regardless of reviewers in subfields like Normative Theory or Comparative Politics being slightly more positive than those in Formal Theory or Methods.Footnote 1 This skewed distribution toward negative recommendations highlights that editors are by no means the only critical source in the editorial process. Note that we invite on average four to five reviewers and base our decision on about two to three reviews per manuscript. To make a fair decision, we read the concerns raised in those reviews instead of just counting the number of positive and negative recommendations. In fact, to fill the pages of a published issue, editors often overrule concerns raised by the reviewers.
FIGURE 1. Share of Reviewer Recommendations by Subfield
However, given different readings of the reviews, there might still be the impression that the decision of the editor was negatively biased. At this point, however, it is important to recall that editors may receive additional private information from the reviewers. In fact, in 40% of cases, reviewers provide such to the editor only. Such as it is, providing the opportunity for sharing private information has its proponents and opponents. Some argue that it increases the risks of publication circles, bias, and unfair comments about manuscripts, whereas others contend that reviewers may fear backlash from accidentally revealing their identity from certain comments or objections they made. Additionally, if subfields are small or the topic is rather narrow, it may be the case that a reviewer has previously seen the paper and raised concerns (e.g., on conferences or with another journal). In this regard, Figure 2 shows the share of reviewers who provide comments to the editors by subfield. The share is largest for Formal Theory, a relatively small and specialized subfield, and is lowest for “American Politics” and “Race, Ethnicity, and Politics,” two rather broadly defined subfields. Although only being suggestive, this pattern is in line with concerns about revealing reviewer’s identity in small or specialized subfields.
FIGURE 2. Share of Reviewers Who Provide a (Private) Comment to the Editor by Subfield
Given the large number of (private) comments to the editors, do they influence editors’ decisions? Put differently, is there any additional information in the comments that change editors’ decision making? A proper analysis of this question is beyond the scope of this short Note, especially given that many comments are only apologies for delayed reports rather than additional elaborations on the feasibility and/or necessity to address concerns raised in the comments to the author. Nevertheless, we can try to shed some suggestive light on this question by comparing the relative share of comments to the editors among reports for which the editors’ decision was in conflict with the general recommendation of the reviewer. If the private comments do not contain any relevant additional information, we would expect the share of comments to be similar to cases when editorial decision and the reviewer recommendation corresponded to each other.
To simplify the analysis, we aggregate recommendations as either positive (Accept & Minor Revision) or negative (Major Revision & Reject). We focus on first decisions by the editor coding them as Reject or Revise and Resubmit (R&R). Table 1 shows that the share of comments to the editor is higher among reviewer-editor combinations that are in conflict (in bold) than among combinations that are aligned. In 47% of cases when the recommendation was negative, but the editor still decided to offer the chance to revise, reviewers also provided a private comment (p = 0.00). In the same regard, the share of comments to the editor was also higher when the recommendation was positive but editors rejected the paper (43%). These are compared with the cases when both were aligned, where only 40% of reviews also came with a private comment (p = 0.10). It suggests that the comments may indeed contain relevant information editors consider when assessing whether authors will be able to revise a manuscript to meet the journal’s standards. Thus, it should be evident that editors who decide not to share this information with the authors to keep reviewers’ confidentiality simultaneously run the risk of raising discontent.
TABLE 1. Share of Reports with a (Private) Comment to the Editor by Recommendation and First Decision
In these “Notes from the Editors,” we wanted to shed some light on the overall distribution and types of reviewer recommendations in the APSR and how they influence editorial decision making. Even though the presented statistics are only explorative and ignore relevant confounders, such as subfield specificities or the level of reviewer disagreement at the manuscript level, we hope they provide some informative insights into the editorial processes at the APSR, thereby promoting trust and understanding among authors (and reviewers) for the decisions our editors take. After all, the APSR relies on the trust and willingness of both authors and reviewers to continue publishing research of exceptional merit in our discipline.
