Scholarly peer review—the idea that the merits of academic work are best judged by subjecting them to the scrutiny of experts from within the relevant field—is central to the twin goals of maintaining standards and providing credibility to published material. Top academic journals in political science regularly receive more than 700 new submissions from graduate students and faculty. According to the most recent editorial reports available on the journal's websites, the Journal of Politics received 729 new submissions in 2010, the American Journal of Political Science received 798 new submissions in 2011, and the American Political Science Review received 685 new submissions in 2010–2011. The editors of these journals recruited, on average, 3,000 political scientists (professors and graduate students)Footnote 1 to assess anonymously these papers' suitability for publication in the specific journal. The completion rate for reviews over this period typically exceeds 50%. For example, AJPS requested 3,196 reviews and 1,770 completed reviews were returned.
Impartial peer review is central to the scientific endeavor that all of us are engaged in and requires a sizable community of expert volunteers to fulfill its promise. However, scholars able and willing to referee manuscripts incur undeniable opportunity costs—the most obvious of which is time that could be spent on research and teaching commitments (Niemi Reference Niemi2006). Although most researchers acknowledge the importance of the peer-review process, they often consider reviewing manuscripts to be a burden—an occupational hazard. As a result, completing reviews often ends up at the bottom of to-do lists.
There are, however, important benefits to consider, and it might be helpful to underscore such benefits before describing the basics of reviewing.
• Refereeing allows you to keep up with cutting-edge research in your sub-field, while also helping to keep your sights set more broadly;
• Too often, we only see the final product, which can give us a false sense of elegance. Reviewing manuscripts in their early stages reminds us that everyone (and every published bit of research) has to go through a process of refinement;
• By exposing you to diverse examples and writing styles, reviewing allows you develop an appreciation of effective writing and helps you improve as a writer;
• Related, refereeing allows you to understand and apply the subtle differences between writing papers for seminars, conferences, and journals;
• Finally, many academic journals allow reviewers to see the other reviews of the same manuscript, which allows you to assess your own review and compare your assessment of a manuscript to what other researchers think about it. It also allows you to see how much disagreement may prevail in evaluating even important manuscripts.
Despite the tangible benefits associated with refereeing manuscripts for publication, graduate programs do not typically advise students on the importance of contributing to the peer-review process or on how to effectively referee manuscripts. In line with recently published articles in other disciplines (Benos, Kirk, and Hall Reference Benos, Kirk and Hall2003; Drotar Reference Drotar2009; Lovejoy, Revenson, and France Reference Lovejoy, Revenson and France2011; Neill Reference Neill2009; Roediger Reference Roediger2007), this article outlines the benefits and responsibilities of peer reviewing for recent, and not so recent, political science researchers. The suggestions and advice provided here represents the collective ideas of the authors, who met as a panel at the 2011 American Political Science Association Annual Meeting. Recent interest in the peer-review process, evident by conference panels on the topic (e.g., APSA's 2011 panel on peer reviewing), articles on the topic (e.g., Niemi Reference Niemi2006) and its discussion on a number of blogs,Footnote 2 raises these questions: What makes a review effective and useful? How does one write such a review?
HOW DO JOURNALS SELECT REFEREES? WHAT SHOULD EDITORS KNOW ABOUT THEIR REFEREES?
Editorial staff at academic journals comb graduate program websites, other published work, and conference registration lists to maintain an ongoing database of potential referees. Each journal has its own policies and guidelines regarding how many reviews they require for each submission as well as whom they will invite to be referees; some journals, such as International Organization or the American Political Science Review, do not reach out to anyone without a PhD.
Within those parameters, when deciding whom they should ask to review a manuscript, journal editors generally try to cover their bases. In other words, a typical review panel will include experts in the field, scholars with general knowledge of the field, and scholars with specific methodological and substantive expertise. If, when an author resubmits a manuscript after making suggested revisions, editors decide the revisions need to be reviewed, they generally ask the same set of reviewers. In a few cases, however, editors do reach out to reviewers beyond the original panel (for instance, when the paper includes a methodological innovation that requires specific expertise, or when previous reviewers have noted their own lack of expert knowledge in some area).
As important as knowing who is asked to serve as a referee is knowing who may not review a manuscript. Most editors try to exclude discussants and chairs of panels where the paper previously has been presented. If their suggestions have been followed, they may be biased in favor; if their suggestions have been disregarded, they may react negatively. Similarly, scholars acknowledged by the author are generally excluded, as are those the author feels have a bias that would prevent an impartial review.
In theory, the review process is “double blind”; none of the reviewers should know who the authors of the manuscript are and the authors should not know who is reviewing their paper. In practice, this is a difficult standard to maintain, and some leading economics and psychology journals have turned to a “single-blind” process. Reviewers who want to identify the author(s) of the manuscript can easily turn to search engines (this is particularly true within more specialized fields). Nonetheless, it is critical that referees approach their review without any bias, or, if they feel they have a slight bias, to acknowledge this to the editor. In those instances where the bias is serious, potential referees should turn down the invitation to review the manuscript. Lastly, reviewers should explicitly note (whether as a part of their review or, separately, in a note to the editor) those cases/issues where they feel they do not have deep expertise (for example, procedural issues in papers about Congress) so that editors can assess their reviews fairly, and where necessary invite additional referees.
WHAT GOES INTO A REVIEW?
In this section we discuss the different components of a “typical” review. Of course, reviewers have their own style and prioritize things differently. That said, making sure that several, if not most, of these elements are included in your review is the best way to insure that it is as helpful to the author(s) as it can be.
Your Judgment of the Quality of the Research/Analysis and the Broader Academic Merits of the Manuscript
When you say there is a strong case for a particular recommendation make a succinct argument as to why. If you think the manuscript has merit, make an argument for why you advocate publication. If you think the manuscript lacks merit, again say why you oppose publication. Editors engage with the submission but they also receive many submissions. Clear, direct, and actionable advice is typically preferred to vague suggestions.Footnote 3 This does not mean you have to be more definitive than you feel comfortable with, but in every case make your argument explicit rather than via a long laundry list followed up by “in summary.”
• Think about whether there is something in the manuscript—the kernel of an idea—which with some work can be brought out. Some editors refer to this as “the angel in the marble.”
• Is it suitable for the audience of the specific journal to which it was submitted? The paper might be salvageable, or even very good, but if the audience of this journal will not read it, then it should not be published there.
• Give two or three foundational points to support your recommendation.
Brief Overview of Where the Manuscript Fits in the Literature or a Debate
Editors will not always know the nuances of the discussion the paper addresses. An extremely brief overview can help set the basis for your suggestions. Providing an overview of how the manuscript engages the literature can also give the author(s) a different perspective on the role of their manuscript.
• Is the contribution empirical, theoretical, or both?
• Compared to previous work, what is the key value-added component?
• Be certain not to conflate new ideas with good, nor assume that revisiting earlier ideas is bad.
Distinguish between Major Deal-Breaking Reservations and Minor Quibbles That You Think Should be Addressed but Which Would Not Sink the Paper if Ignored
It is reasonable to make a broad variety of suggestions, but be clear about the importance of different types of suggestions.
• Important but usually secondary suggestions include issues of formatting, readability, typos, and suggestions for extensions.
• Do not hold a manuscript up with a suggested extension (e.g., further research or experiments) unless you see it as central to the contribution of the paper. Let the authors decide.
• If there are missing citations that are not essential but desirable, probably raise them toward the end of the review.
HOW SHOULD YOU WRITE A GOOD REVIEW?
A good review—one that helps journal editors make an informed decision about whether to publish a paper—is not merely one that covers the various components from the previous section. Certainly being comprehensive is important, but it is not what separates a good review from a passable one. In what follows, we discuss ways that will help you write an effective and helpful review and earn the respect of editors, authors, and just might earn you a “Super Reviewer” badge at the next academic conference you attend.Footnote 4
Take the Process Seriously
If a paper gets sent out for peer review, it has survived the editor's desk review and, therefore, has presumably at least some redeeming quality. Refereeing is a key part of your professional responsibility and not a chore toward which you dedicate minimal time and cognitive effort. And, as already noted, there is something in it for you, too.
Be Timely
If you know you will not be able to get to the review for the next several weeks, it is your responsibility to let the editor know. Tardy reviewers are the single most important bottleneck in the publication process, and without being too dramatic about it, you might be jeopardizing a colleague's career. Think about it. Your (negative) review, sent to the editor three months after you accepted the responsibility, means that the author will have had to wait four or five months to find out he or she will not get published in the journal and now will have to go through the same process with another journal.Footnote 5 And tenure review starts in six months.
Be Realistic
Often you are under a time crunch and may not be able to complete the review. If so, decline to do this particular review, but offer one or two suggestions about others who you feel may be well qualified. However, do not get into a habit of routinely declining to review. Editors have long memories.
Be Succinct, but Thorough
An average review should be a couple of pages long. Also, organize the review in some way, such as distinguishing between theoretical and empirical issues, and major and minor suggestions. If, exceptionally, you think the paper is completely unsuitable for the specific journal, be sure to specify why you believe this. If it is suitable for another journal, mention an example or two of a more appropriate venue.
Be Courteous and Gracious
There may be a place in life for snide comments; a review of a manuscript is definitely not it. Good scholarship is a process that often involves submitting imperfect ideas to scrutiny. It is easier to be critical than to be constructive. Acknowledge the author's effort and try to highlight the paper's strengths so that they can be showcased, even if you clearly recommend rejection. And remember, your own scholarship will also be subject to the same scrutiny.
Be Specific and Avoid Criticisms without Context
Broad generalizations—for instance, claiming an experimental research design “has no external validity” or merely stating “the literature review is incomplete”—are unhelpful. Take the time and effort to qualify and explain those statements. At the same time, remember that it is not your paper and, as a reviewer, it is not your job to fix it.
Appreciate the Evolutionary Nature of Research
Often tradeoffs exist between theoretical and analytical strengths (sound identification/causal analysis, etc.) and empirical strengths (good datasets/strong observational data, etc.). When deciding how to assess those tradeoffs, remember that we exist in a marketplace of ideas, and the scientific process is an evolutionary one that involves ongoing improvement (rather than first-time perfection). Often a good paper opens a useful debate rather than settling it once and for all.
WHAT ELSE SHOULD YOU KNOW ABOUT REFEREEING MANUSCRIPTS FOR ACADEMIC JOURNALS?
Academic journals frequently require referees to supplement their written review with a quantitative evaluation of the manuscript. This quantitative section might ask you to assess, for instance, the importance of the topic, the clarity of the argument, or the persuasiveness of the evidence. Each journal has a slightly different set of questions, and a different (often arbitrary) scale on which they should be assessed. In other words, if the responses seem subjective to you, it is because they are. Keep in mind that what is most important about this section is not the specific number you assign to each answer. Rather, editors ask these questions because they underscore the parameters—relevance, clarity, persuasiveness, originality, writing—that should frame your review. Therefore, rather than agonizing over whether the paper you are reviewing is a 3 or a 4 on any given question, make sure you cover all the questions in your written review.
When refereeing a manuscript, your main responsibility is to give the journal's editorial staff advice about whether it should be published in that specific journal, and why. In practice, what this implies is that you need to make a substantiated argument about the scholarly merits of the paper while keeping in mind the journal's purpose and audience. Also, to the extent that there are any tradeoffs between arguing the (de)merits of the paper versus general comments to improve it, remember that the paper is not yours to fix. The specific decision—accept, reject, or revise and resubmit—is also not yours to make, so whereas you should feel free to suggest a course of action, do not feel compelled to.
Along the same lines, it is always useful to visit the journal's website and educate yourself about its purpose, audience, and manuscript acceptance rate. This will clarify the journal's publication criteria, which in turn allows you, as a reviewer, to offer a more considered and effective opinion. Comparing your reviews to the journal's acceptance rate will also help you calibrate your own standards as a reviewer; for instance, if you have rejected all five articles you were asked to review for a journal that typically publishes 50% of its submissions, you are probably being too critical in your assessment. If you have recommended acceptance or minor revisions on all five articles sent to you by a journal that typically publishes only 10% of submissions, you are being too lenient (and will probably be invited less often).
Visiting journal websites will also help you assess more accurately which journals would be most receptive to publishing your own research, thereby saving you the anxiety of waiting several months only to find that the journal thinks that your paper does not fit or is not of high enough quality for its readership. In fact, of the 15% to 30% (depending on journal) of manuscripts that get “desk rejected” (in other words, the journal's editor does not even send them out to reviewers), many are simply deemed inappropriate for the journal's target audience. Manuscripts whose topics do not fall within the scope of the journal will never be published there, no matter how good they are per se.
Finally, to reiterate an earlier point—namely, to take the review process seriously—we would remind would-be referees of the following. Although you are not, strictly speaking, under review yourself, you have every incentive to do at least due diligence. On the one hand, a thorough, persuasive review will not only earn you respect and a reputation, you might even be invited to publish a formal response to the original article in the same journal. On the other hand, a particularly shoddy review will often elicit an apology on your behalf from the journal's editor to the manuscript's author, acknowledging that yours was a particularly superficial and unhelpful review. Further, the memo that is sent by the editor to the author is also sent to you and the other referees. The confidential nature of the review process means that you escape with your dignity somewhat intact, but only just!
NOW WHAT?
With those guidelines out of the way, that leaves one last question: “How do I sign up?” Some journals, especially subfield journals, encourage would-be reviewers to e-mail the journal's managing editor to volunteer. Other journals, like AJPS and APSR, will approach you (subject to their specific policies), based on the work you have published or presented at conferences. The longer answer is that, although in general the journals will approach you (subject to their specific policies), you should always feel free to contact them directly if you are interested.
ACKNOWLEDGMENTS
Our thanks to Vinay Jawahar for editorial assistance and Sean Twombly for organizational assistance. This article came out of a 2011 APSA panel on refereeing organized by Dustin Tingley.
