In 2013, Sheryl Sandberg ignited a global conversation about what women could do to increase their chances of becoming leaders in their chosen fields (Sandberg Reference Sandberg2013). Sandberg’s “lean in” mantra was attractive because it suggested that by claiming a place at the table, and demonstrating one’s potential, women were more likely to be recognized and rewarded. Her argument was criticized from many angles. Relevant for this study were the criticisms that Sandberg was diminishing the importance of structural factors and downplaying the role of implicit bias (Eddy and Ward Reference Eddy, Kelly, Eddy, Ward and Khwaja2017, Kossek, Su, and Wu Reference Kossek, Su and Wu2016). Women could lean in, this criticism suggested, but leaning in did not necessarily translate into greater recognition. Our paper, part of a larger study investigating gender and the construction and projection of status within the discipline of American political science, Footnote 1 examines this criticism in the context of political science.
Our study focuses on a set of institutions that connote and construct professional status. We pay special attention to gender differences, how women might be leaning in, and whether these efforts translate into higher-level status recognition. Considering how status is constructed within the discipline of political science has been difficult. Ostensibly, status recognition in academia rewards research excellence, and sometimes also excellence in mentoring, teaching, and leadership. Yet there is no agreed upon metric of excellence, and most proxies reflect implicit subfield, method, topic, and probably also gender and race biases. It is also difficult to collect information for comparison, and thereby to assess relative merit for an award. Past studies relied on surveys of scholars and citation counts to assess status. We focus on election or selection to peak institutions within our discipline, including appointments in research departments, leadership in top journals and professional associations, and honorific recognitions such as winning important prizes, being asked to give a named plenary address, and appointments to honor societies such as the American Academy of Arts and Sciences. We also consider the accumulation of status recognition and compare across different types of status recognition. Our focus is always on gender differences.
Since it is hard for any selection process to systematically compare the accomplishments of candidates, we can hardly criticize selection methods that rely on self, networked, or loosely defined status assessments. We therefore investigate the one remedy for gender bias that we could imagine: accountability. Where gender balances are monitored and selectors are held accountable for gender disparities, we might reasonably expect a greater gender parity.
Our analysis investigates three interrelated questions: 1) Do institutions within the discipline of political science—including departments, the American Political Science Association (APSA), editorial boards, and academic honor societies—reflect or remedy gender disparities that exist in many forms of recognition, including appointments to top leadership and citations? 2) Are institutions with centralized and accountable appointment mechanisms less gender skewed compared to more networked and decentralized selection processes where implicit bias may go unchecked? 3) Does leaning in help? Does the effort of women to publish and to claim a seat at leadership tables increase the likelihood that higher-level status positions will follow?
We will answer these questions, but perhaps the strongest finding is that women are leaning in. Examining the various ways men and women assume leadership in our discipline, we show that women are stepping up to lead, and they are leaning in even more than men. But the result of this stepping up seems to be that women are doing perhaps more than their fair share of high-service/low-recognition leadership, and this leadership is not leading to higher-status recognition.
A primary aim of our analysis is to provide an accurate picture of status recognition in peak institutions of American political science, so that we can understand how and whether there is a difference in how men and women contribute, lead, and get recognized. A sad and challenging baseline emerged in this analysis: women comprise 28% of tenured political science faculty. Well-meaning leaders often seek greater gender parity. Yet the 28% baseline means that when women comprise 30% or more of a journal’s editorial board, reviewer pool, or association section heads and council members, women are probably being over serviced. Meanwhile, in 2020, who can be happy with a goal of 30% female representation, let alone a finding that women comprise significantly less than 30% of the top positions—full professors, editors of journals, speakers for named events, or top leaders in academic associations? This sad baseline generates a damned-if-you-do/damned-if-you-don’t dilemma. Either we aim for no more than a 30% representation of women to avoid over-servicing female academics, or we as a discipline decide to value leaning in, and thus to honor and select for our highest honors those who are leading, contributing, and distinguishing themselves based on a broader definition of contributions to our profession. Rewarding leadership in its many forms may lead men to then value service-heavy leadership more. Meanwhile, without this change scholarly publication—and especially publication on fashionable topics—will count more, in which case men (and women) who publish instead of taking up less-rewarded or less-prestigious leadership roles may be doubly rewarded.
First we review what we already know about gender disparities in recognition in academia, and within the discipline of American political science. Next we explain our methodology of creating two datasets, one that defines a baseline for evaluating gender representation, and a second that assesses the gender representation in peak institutions within American political science. We then report on our findings regarding gender distribution within top institutions in our discipline, including academic associations, editorial boards, and honor societies. This material allows us to explore the lean-in expectation that claiming a seat at the table by being selected to gateway status positions and publishing will contribute to higher levels of status recognition, and how accountability affects gender representation in status positions. The conclusion addresses the three questions posed in this introduction.
Assessing Status in the Profession of American Political Science
How do we as political scientists assess status? Early studies relied on surveys of the American political science community. In 1964, Somit and Tanenhaus’ comprehensive study found that the road to individual success was determined by the quantity and quality of publications, doctoral pedigree, having the right connections, gaining research support, procurement of funding, textbook authoring, teaching ability, and of course, chance (Somit and Tanenhaus Reference Somit and Joseph1964, 79). Meanwhile scholarly eminence came from developing new analytical tools and concepts, systematizing new fields in the discipline, catalyzing new intellectual movements, and providing leading service in the discipline (Somit and Tanenhaus Reference Somit and Joseph1964, 67-9). Roettger replicated Somit and Tanenhaus’ study in 1978, comparing both to an “outsiders” view of “prestigious contemporary American intellectuals.” Roettger found that the criteria political scientists used meant that the production of status and notability involved a relatively small group of “taste-makers and standard setters” contributing “in a disproportionate fashion to the definition of professional excellence. As a result, [these elites] are, to a large extent, able to perpetuate their own pre-eminence and contribute to that of their students” (Roettger Reference Roettger1978, 10).
These studies were criticized for their reliance on surveys, a method many saw as little more than a popularity contest. The search for more objective measures involved creating models, looking at citation rates, and analyzing the prestige and placement records of departments (Burris Reference Burris2004; Robey Reference Robey1982; Bingham and Vertz Reference Bingham and Vertz1983; Klingemann Reference Klingemann1986; Fowler, Grofman, and Masuoka Reference Fowler, Grofman and Masuoka2007). These approaches to some extent replicated the clubby tastemaker nature of status. Meanwhile, the discipline as a whole disliked how status was manufactured. Drejmanis’ sweeping multi-source study of American political scientists notes a resigned recognition combined with a dissatisfaction of the status quo orientation of the discipline, the primacy of behaviorialist methodology, the many studies of trivial significance, and the failure of political scientists to speak truth to power. Drejmanis concludes that “all in all, American political science, after a century of almost continuous growth, finds itself disciplinarily and professionally demoralized. External forces are creating profound changes, the shape of which is only dimly perceived” (Dreijmanis Reference Dreijmanis1983, 217). These findings parallel what Somit and Tanenhaus had documented twenty years earlier (Somit and Tanenhaus Reference Somit and Joseph1964, chap. 2).
Today the extent to which these early studies ignored gender seems remarkable. Recognizing that every high-prestige individual was a man, Somit and Tanenhaus noted that “big men” repeatedly reoccur on status lists of scholars, yet they did not investigate why this was so. Dreijmanis did not ask how gender or diversity, or a lack thereof, might contribute to his findings (Dreijmanis Reference Dreijmanis1983, 209-11). Only in the American Political Science Association’s Committee on the Status of Women in the Profession were gender (and later race) dynamics in the profession openly studied and discussed (Shames Reference Shames and O’Connor2010; Sedowski Reference Sedowski2007; Sedowski and Brintall Reference Sedowski and Brintall2007; Brandes et al. Reference Brandes, Buker, Burgess, Cook, Flammang, Geiger, Okin, Yoon and Ackelsberg2001).
Also remarkable is the disconnect between gender-blind studies and scholarship that does focus on gender. In the business world, studies find that gender and racial diversity contribute to “smarter teams” with measurably improved results (Rock and Grant Reference Rock and Grant2016; Neale, Northcraft, and Jehn Reference Neale, Northcraft and Jehn1999). Yet across academic disciplines studies have found gender gaps in publication and productivity levels, citations, invitations to speak at important meetings, prize awards, prize remuneration, grant awards, patents and more (e.g., Oliveira et al. Reference Oliveira, Ma, Woodruff and Uzzi2019; Ma et al. Reference Ma, Oliveira, Woodruff and Uzzi2019; Ferber and Brün Reference Ferber and Brün.2011; Elsevier 2017; Zuckerman et al. Reference Zuckerman, Cole and Bruer1991; Hesli and Lee Reference Hesli and Lee.2011; Dion, Sumner, and Mitchell Reference Dion, Lawrence Sumner and Mitchell2018; Nittrouer et al. Reference Nittrouer, Hebl, Ashburn-Nardo, Trump-Steele, Lane and Valian2018). Meanwhile, studies also find that female scholars have similar research and output aspirations as men, and there are no clear gender differences in their ability to innovate or contribute (Elsevier 2017; Zuckerman et al. Reference Zuckerman, Cole and Bruer1991). The implication is that academia is gender skewed in ways that undermine output, creativity, and recognition.
These trends also exist in political science, where studies have documented that women publish less in general and in the profession’s top journals (Breuning and Sanders Reference Breuning and Sanders2007; Young Reference Young1995; Teele and Thelen Reference Teele and Thelen2017; Samuels and Teele Reference Samuels and Langan Teele.2018; Key and Sumner Reference Key and Lawrence Sumner.2019); that women often study different topics and the subjects women study are less valorized compared to the subjects men study (Maliniak, Oakes, and Tierney Reference Maliniak, Oakes and Tierney2008, 123; Leahey Reference Leahey2007; Shames and Wise Reference Shames and Wise.2017; Key and Sumner Reference Key and Lawrence Sumner.2019); that top journals more often publish topics that interest men compared to women (Key and Sumner Reference Key and Lawrence Sumner.2019); that women are cited less when they do publish (Maliniak, Powers, and Walter Reference Maliniak, Ryan and Walter2013; Kim and Grofman Reference Kim and Grofman2019a; Dion, Sumner, and Mitchell Reference Dion, Lawrence Sumner and Mitchell2018); that the gender patterns found in articles (lower publication rates and fewer citations) are even worse with respect to book publishing (Samuels and Teele Reference Samuels and Langan Teele.2018); that women are less likely to be lead authors (Evans and Moulder Reference Evans and Moulder2011); that co-authorship boosts submission rates more for men compared to women (Djupe, Smith, and Sokhey Reference Djupe, Smith and Sokhey2019); that female scholarship is less likely to appear on graduate syllabi (Colgan Reference Colgan2016; Phull, Ciflikli, and Meibauer Reference Phull, Ciflikli and Meibauer2018); women receive lower course evaluations compared to men (Martin Reference Martin2016); that “women, although they win more awards today than previously, still are unable to match men in the scholarly recognition given their work” (Tatalovich and Frendreis Reference Tatalovich and Frendreis2018, 8); and that these findings are longstanding, replicated across vast stretches of time (Young Reference Young1995; Mathews and Andersen Reference Mathews and Andersen.2001; Breuning and Sanders Reference Breuning and Sanders2007).
There is no single explanation for why gender differences are found in nearly every indicator, or why these differences persist notwithstanding efforts to promote gender equity. In the natural sciences, scholars have identified institutional and structural explanations such as social exclusion, diminished collaboration opportunities, difference in resources (e.g., lab funding, grants, lab size) and individualistic and behavioral differences including gendered “scientific and professional styles” (Sonnert and Holton Reference Sonnert and Holton1996, 66, 67). Scholars have noted that a yardstick rewarding sameness, especially if this yardstick favors men (e.g., publication and citation rates, or working on highly competitive topics) perpetuates male status-dominance (Keller Reference Keller, Zuckerman, Cole and Bruer1991, 233-4), meanwhile structural exclusion and gender dynamics can render similar female contributions less visible or valued (Epstein Reference Epstein, Zuckerman, Cole and Bruer1991, 249-51, Sonnert and Holton Reference Sonnert and Holton1996, 67-69).
These ideas come together in the kick-reaction thesis, which is Cole and Singer’s explanation for why women in science are less productive (as measured by publication rates) (Cole and Singer Reference Cole, Singer., Zuckerman, Cole and Bruer1991). Cole and Singer note that at the early stages of scientific careers, women and men achieve similar levels of distinction in terms of acceptance rates to top institutions, receipt of post-doctoral fellowships, positive grant and publication decisions, and early career honorifics. This finding belies the notion that initial conditions (e.g., pedigree and graduate/post-doctoral success) and inherent or endemic factors explain the gender disparities in productivity across a career. The authors investigate the effects of positive and negative feedback—what they call small “kicks” that can speed or slow a scholar’s future productivity. They find that over the course of a career, these singularly limited impact kicks have a cumulative effect, generating the findings of gender differences in output (Cole and Singer Reference Cole, Singer., Zuckerman, Cole and Bruer1991, 283). Picking up this idea, Sonnert and Holton document that more negative feedback contributes to the finding that women scientists who were high achievers at the start of their career are more likely to see themselves as average in ability, and more likely to say that they handled career obstacles with less confidence and assertiveness. This difference, they argue, may explain in part why, as a group, across every metric women scientists produce and achieve less than their male counterparts (Sonnert and Holton Reference Sonnert and Holton1996, 67).
While we cannot test the kick-reaction thesis, there is much to suggest that it applies to political science. We already mentioned the parade of negative kicks for women including lower citation rates, less prevalence on graduate syllabi, lower valuing of topics that interest women, lower teaching evaluations, etc. Recent studies suggest that these negative kicks affect women’s submission and publication practices. Observing that when women submit work to top journals, they publish at similar if not better rates compared to men, Djupe, Smith, and Sokhey nonetheless find that men are far more likely to submit articles to top journals. This difference is partly strategic—women are more likely to submit to journals where they expect to be published, whereas male scholars (particularly assistant professors) have no qualms about “flooding” top journals with submissions that then garner a high numbers of rejections (Djupe, Smith, and Sokhey Reference Djupe, Smith and Sokhey2019, 74). But there is also a more prevalent presumption that top journals will not be interested in female scholars’ work (Djupe, Smith, and Sokhey Reference Djupe, Smith and Sokhey2019, 74), a presumption that is not wholly without merit given the publication patterns of top journals (Key and Sumner Reference Key and Lawrence Sumner.2019).
We offer a new way to examine gender and status in our discipline, focusing on the selection by peers to status positions in peak institutions, where selection in some way recognizes past accomplishments. We did not set out to study service, nor do we code service awards. Yet we came to realize that the different status metrics encapsulate different types of recognition and levels of service. We thus found a troubling relationship between service and status. To gain status, young scholars are encouraged to develop their social networks and produce prize-recognized scholarship. The high-service/low-status types of appointments we code promote these goals. Becoming a leader within prestige-peak institutions builds networks, which in turn can lead to invitations to speak and to job offers (Guarino and Borden Reference Guarino and Borden2017, 673). International Studies Association (ISA) or APSA sections build social networks, prizes for underrecognized subfields, and presentation slots at flagship conferences. We explore whether these low-status/high-service positions translate into greater individual status recognition.
Constructing Status in American Political Science: Our Datasets
The two datasets we built for this study—a Baseline and a Status dataset—are both shaped by selection decisions of political scientists. They are therefore indicators of peer recognition within peak institutions in American political science, simultaneously reflecting and constituting status recognition. We code entries according to a scholar’s name, allowing us to trace and compare rates of men and women achieving higher-level status recognition. Later we bring in named data from Teele and Thelen’s study on publishing in top ten political science journals (Teele and Thelen Reference Teele and Thelen2017), and Kim and Grofman’s top 400 active political scientists, as measured by citation counts (Kim and Grofman Reference Kim and Grofman2019a). Online appendix 1 explains coding for all four data sources.
The Baseline Dataset: How Research Universities Construct Status
Since we seek to capture status, we constructed a Baseline dataset that includes faculty members in U.S. universities that confer doctorates, drawing on the Carnegie higher-education classification system of doctoral granting research universities, which is divided into three groups: Research University/Very High level (RU/VH, N= 115); Research University with High Levels (RU/H, N=107); and Doctoral Research institutions (DRU, N=112). Footnote 2 We visited the websites of political science departments (including departments with alternate titles, such as Government, Politics, etc.), noting the faculty member’s name and rank (assistant, associate, professor; non-tenure categories such as research, professor of practice or lecturer, and emeritus professors). We coded gender based on pronouns in the faculty member’s bio or their profile picture. Footnote 3 From the 334 institutions, we catalogued 6,696 individuals with appointments in political science departments in 2016.
We use the Baseline dataset to generate expectations for gender representation and to explore if men and women are equally likely to appear across our status indicators. Since the Baseline dataset captures a single year, we can capture academic rank and institution type for each individual. Our reliance on the Carnegie higher-education classification, however, means that our Baseline dataset excludes American political scientists working abroad, political scientists at U.S. institutions that do not grant PhDs, and scholars with a primary appointment in a cognate department (e.g., sociology, international studies) or at a policy school; our Status dataset is not limited in this way. As a check, we compared our Baseline data to aggregated APSA membership data which is not limited to PhD granting institutions (refer to online appendix 2). Gender differences are small; our Baseline dataset has slightly higher percentages of female assistant, associate, professor, and non-tenure-track faculty. Given that assistant professors and non-tenure line faculty comprise a larger proportion of APSA membership, our Baseline dataset arguably provides a more representative picture of tenured faculty.
Figure 1 reports our Baseline data for each rank. Even considering that there are slightly more RU/VH institutions compared to the RU/H and DRU categories (115 versus 107 versus 112), we can see that RU/VH institutions have significantly larger political science faculty. Figure 1 also shows that RU/VH institutions are relatively top-heavy; full and emeritus faculty outnumber assistant plus associate professors.
Figure 2 reports on the gender distribution within each rank in our Baseline dataset. Most of the status indicators draw on tenured faculty, thus our gender balance status expectations combine active tenured faculty (associate and full professors, but not emeritus faculty). This generates a baseline expectation of 28% female for our status indicators, which we generally denote by a dashed line. Any indicator where the gender representation is lower than this 28% expectation reflects a gender imbalance. Meanwhile, for the highest levels of status, a 23.3% baseline (the percent of female full professors) may be more accurate.
Figure 2 indicates a comparatively large proportion of women at the assistant rank (41.8%)—a promising sign for the future, although the percent of women faculty grows slowly. Footnote 4 We also compared within the RU/VH, RU/H, and DRU categories, finding similar gender distributions in each category with the exception of the assistant professor level, where there were more females at RU/VH (41.9%), RU/H (44.2%), compared to DRU (36.6%) institutions and APSA membership (38.7%).
The Status Dataset: How the Profession Constructs Status
The Status dataset provides a heretofore unknown glimpse into gender differences in professional association leadership, editorial boards, and honorific appointments, appointments that signify leadership and eminence. Since we are coding the positions themselves, the dataset includes political scientists based at universities outside of the U.S., non-PhD-granting institutions, and emeritus and even deceased faculty, but it does not capture faculty rank. We continue to draw on the assumption generated from the Baseline dataset that 28% female means that the status indicator is gender representative. This baseline assumption makes sense because most of the status positions we coded require or presume tenure.
Each position in our Status dataset (e.g. editorial board, APSA Council, named lecturer, member of an academic honor society) is counted separately. Footnote 5 We discuss individual political scientists as “names.” Since each name can have more than one status appointment, we give the term “observation” to reflect each coding within a status category. Individuals “accrue status” by accumulating recognition across indicators. Our Status dataset includes 4,990 observations of 2,648 named political scientists. In terms of names, females comprise 31.3% of the Status dataset (N=830) and men comprise 68.7% (N=1,818). If we presume that seniority in the form of tenure increases representation in status positions, then overall our Status dataset largely reflects the baseline of gender in the discipline.
The various categories included in our Status dataset, however, are not equal to each other. For example, being a section head in APSA or a member of an editorial board does not connote the same level of status as president of APSA, or a member of the American Academy of Arts and Sciences. The next section explains how and why we see each coded item as indicating a form of status, and how we weight and use the different categories.
Gender Distribution and Recognition within Different American Political Science Institutions
The journals, academic associations, and honor societies we code are peak institutions within political science. Footnote 6 The status of these institutions comes in part from who is listed on a masthead, and being on a masthead also contributes to an individual’s status. Since every member on the masthead has been selected by peers, these appointments are also a form of status recognition. This section explains why we associate status with the positions, and how we weighted the different coding categories. We then describe the gender distribution of faculty within each weighted category and across different institutions within American political science.
Accruing Status: Categories and Weighting in the Status Dataset
We wanted to explore whether individuals could accrue status by leaning in, serving in gateway leadership positions. We will also consider more scholarly forms of leaning in such as publishing in top journals and working at an RU/VH institution. We created the general terms “Committee,” “Officer,” “Leader,” and “Honor” to aggregate and reflect our understandings of particular categories of leadership and status-recognition associated with a particular appointment, weighting these categories differently.
Committee positions recognize the past achievements of an individual, such as publishing in a top journal or being known enough to win election to an office (e.g., section head or the Council of a professional association). Yet these positions generally require a fair amount of service and they offer only limited individualized benefits, namely social networking opportunities and the potential to influence collective decisions. Since committee members are appointed based on past achievements; because committee members are entering the room where decisions are made and becoming part of a leadership network; and since committee members are listed in mastheads and becoming more well known, a committee position arguably confers status. We recognize that section leadership is not fully analogous to being elected to the APSA Council. Both are given a weight of 1, but throughout we consider section leadership separately.
Officer positions are akin to middle-level management. They involve more responsibility but do not confer on the individual the greatest rewards of leadership and honor. Officer positions generally draw from committee members, and in principle officer positions should be earned based on prudential judgement shown as a committee member. Since many decisions are made by leadership teams (Officers + Leaders), and new leaders are often selected from the officer pool, we gave the Officer position a higher weight (2).
Leader positions are fewer in number, generally involve service, but also recognize past leadership and scholarly recognition. Being president of APSA will involve meetings with top people in the profession, higher visibility, and being asked by others (e.g., the Nobel Prize committee, the Johan Skytte and Carnegie selection committees, U.S. politicians seeking expert advice) to nominate, advise, join selection bodies, or otherwise help select future leaders in the discipline. In this way, leadership positions provide important gatekeeping roles. Leadership positions may also be observed by head-hunters looking for talent. Since leadership positions both reflect and construct high levels of recognition, we give a higher weight (3) to a leader position.
Honor positions are pure status positions that are, in principle, based entirely on scholarly achievements. While the recipient may be asked to give a talk, honor positions involve little to no service. Faculty recognized by an honor position tend to be given endowed chairs, are paid more, and are hired away by status-seeking institutions. Because of the difference in service, we separate Honor and Leadership categories but award both the highest weighting (3).
Table 1 summarizes our status categories and weighting. The number of observations is greater than the number of names because many people have multiple status appointments.
Source: Alter et al. Reference Alter, Jean, Schraudenbach and Rozier2020
* Section leadership includes President/Chair, Vice President/Chair, Treasurer, and Secretary. 775 of the Section officer names (29% of the total names) appear in the Status dataset only by virtue of section leadership. Refer to online appendix 1 for more on the section leadership data.
How Men and Women Accrue Status
The categorization and weighting provide one way to assess whether women are leaning in, showing up, and seeking both leadership and recognition. Our finding is somewhat dispiriting. Yes, women are showing up, claiming a seat at the table, and providing leadership. But they are showing up at the most service-heavy level, and the next section will show that this leaning in does not translate to higher-level appointments. Figure 3 reports our findings based on gender and weighted status coding, drawing on observations, which are more than 50% larger than the coded names. Because section leadership is a large and gender-skewed part of the Committee category, we report the Section Officer and Council/Editorial board data separately. Using our 28% baseline, we can see that women are over-represented in the Committee and Officer categories, and under-represented in the Leader and Honor categories.
Figure 3 documents that the share of women in the lowest weighted and relatively service-heavy status roles is substantially higher than what our baseline expects (36.5% of weighted 1 and 38% of the weighted 2 status positions are held by women, compared to our 28% baseline presumption). To investigate further, we considered the average number of times men and women appeared in our Status coding. When we exclude section leadership, men are more present in our Status dataset (e.g., have relatively more observations) compared to women. Yet when we include section leadership, women become more present in our Status dataset (e.g., have relatively more observations) compared to men. We probe this difference further in the statistical analysis that follows, and in online appendix 4.
The greatest glass ceiling exists, unsurprisingly, at the very top. Table 2 reports the overall breakdowns in our Leadership and Honor (weighted level 3) categories. Because top leadership and honor positions tend to be conferred only on full professors, we lower the baseline to 23.3%. Women appear to be under-represented within honor societies, named prizes and lecturers, and among APSA and ISA presidents. We worried that a time-lag might explain why these top honors remain male dominated. Studies of citation patterns and our findings for the Editor category suggest otherwise. Footnote 7 Female representation has improved. Table 2’s third column repeats the analysis drawing on the earliest records through 2000. We can see that gender balance has improved over time.
Source: Alter et al. Reference Alter, Jean, Schraudenbach and Rozier2020
The baseline expectation is 23.3%, since these positions are generally draw from full professors. For additional gender breakdowns for Honor and Leader categories, refer to online appendix 1.
Figure 4 examines gender balances across the various status categories (figure 6 does the same with respect to journals). It, too, demonstrates that women are overrepresented at the lower-level/more service-heavy types of leadership.
A Special Focus on Journals
The topic of gender in political science journals has been widely studied. Scholars have found gender disparities in editorial boards, publication and citation rates, topics that are published, and more (Maliniak, Oakes, and Tierney Reference Maliniak, Oakes and Tierney2008; Maliniak, Powers, and Walter Reference Maliniak, Ryan and Walter2013; Mitchell, Lange, and Brus Reference Mitchell, Lange and Brus2013; Tatalovich and Frendreis Reference Tatalovich and Frendreis2018; Teele and Thelen Reference Teele and Thelen2017; Dion, Sumner, and Mitchell Reference Dion, Lawrence Sumner and Mitchell2018; Key and Sumner Reference Key and Lawrence Sumner.2019). We conducted a longitudinal coding of editorial boards from 2000–2015 across fifteen journals. Figure 5 reports on our coding, including all board members for each listed journal during the coding period. Our baseline remains 28%, since mostly tenured faculty serve on editorial boards. Some journals, for example Comparative Politics, Political Theory, and International Security, have had very little board turnover (indicated by the lower number of names). Others, such as APSR, have larger boards and more turnover.
Online appendix 3 works with the Teele and Thelen data, comparing the gender composition of editorial boards and publication in specific journals. The analysis suggests that some journals failed to view female scholars publishing in their journals as a potential leadership pool, thus Comparative Politics, Comparative Political Studies, International Security, and Journal of Conflict Resolution historically under-represented women compared to rates of female publishing, while Political Theory over-represented women on its editorial board. Online appendix 3 also compares our longitudinal data to 2017 data, to see if the current focus on gender in the journals has shifted the composition of editorial leadership. Every journal, with the exception of International Security, increased women’s representation, in some cases quite significantly. Yet the course-correction also contributed to over representation, and thus over-servicing, of women.
A final view, figure 6, examines the gender composition across editorial boards, associate editors, and editors. Women are present at middle ranks on par or exceeding our baseline expectations, and female representation is greater at the lower compared to top leadership (31% of editorial boards representation, compared to 36% of associate editors, and 22% of editors). This comparison incorporating multiple years produces results that are slightly less gender skewed compared Stegmaier, Palmer, and van Assendelft, who found that in 2010 women comprised only 18% of editors, 23% of editors plus associate editors, and 26% of editorial board members (Stegmaier, Palmer, and van Assendelft Reference Stegmaier, Palmer and van Assendelft2011, 801, table 2).
Given the lack of clear evidence that the gender of reviewers influences aggregate review outcomes, Footnote 8 perhaps we need not worry about how gender compositions affect journal publications. Yet diversity in its many forms can affect submissions to journals (Brown and Samuels Reference Brown and Samuels.2018; Djupe, Smith, and Sokhey Reference Djupe, Smith and Sokhey2019; Brown et al. 2018). Since editorial boards can help to set direction, monitor reports of the editor, provide a pool of future editors and peer reviewers, and thus serve as gate-keepers, we wanted to know how editors thought about diversity on their board, and about the contributions and uses of their editorial boards. We report on our qualitative investigations in online appendix 3. A key takeaway is that although editorial boards could be used to change submissions patterns and publication topics, this has not yet occurred. The recent decision to appoint a team of twelve female editors for APSR may address this omission. Footnote 9
Overall, this section has documented that women are present in status positions, at times in proportion to the baseline. Yet women are more present, and perhaps even over-represented, at the committee and the officer levels and are under-represented at the highest levels of leadership and honor recognition. The next section will investigate whether high-service/lower-status leadership is a gateway to higher-level status appointments.
Exploring Sandberg’s Lean-in Thesis in the Context of American Political Science
Here we examine whether leaning in contributes to status. We analyze two forms of leaning in—leadership and publishing. Our initial focus was masthead leadership—editorial boards of top journals, APSA and ISA Council leadership, and mid-level officer appointments in journals and academic associations. Participation at these lower leadership levels can be seen as gateway efforts to rise into the Leader/Honor category. We then added section leadership data provided by APSA and ISA. Section leadership also helps build a scholar’s profile within an important social network—scholars working in their area who might cite their work or write a tenure or promotion letter. APSA also draws on section leaders for nominations to higher-level appointments. Finally, we brought in the Teele and Thelen data on journal publications, and for the accrued status analysis we draw on Kim and Grofman’s data on the top 400 citation earners. As always, we are interested in gender differences across status indicators.
Since we are tracking individuals, we first need to know the extent to which our various data sources overlap. Figure 7 captures the overlap across the datasets based on unique names, and where overlapping categories indicate a name shared across datasets. There is less overlap with the Teele and Thelen journal publishing data, since publication data captures graduate students and authors not based at U.S. PhD-granting institutions. There is more overlap with the Kim and Grofman data of the top 400 citation earners, since both datasets code political science faculty at U.S. PhD-granting institutions.
Who Is Missing? Who Is Present?
We began with an intent to replicate Reagle and Rhue’s study (Reference Reagle and Rhue2011) exploring which notable scholars were missing from Wikipedia as compared to Encyclopedia Britannica. Our study evolved, yet it is still helpful to use the technique of exploring whether men and women are “missing” at equal rates across our indictors. The phrasing is a bit awkward, but basically we ask whether women are more likely than men to be absent in key status categories. Lower numbers are better, because a lower number means that fewer men/women are missing from a category. We also ask whether knowing that a scholar is a man helps predict whether this scholar will appear in a status category.
When we consider tenure-line faculty from our Baseline dataset, and exclude section leadership, we find that over 80% of Baseline faculty are not present in the Status dataset, suggesting that we are capturing status. We also find that men and women are “missing” at comparable rates, which means that there are no appreciable gender differences. When focusing on tenure-line faculty at RU/VH institutions, again excluding section leadership, women were relatively less likely to be missing. Table 3 reports these findings, demonstrating a higher overlap between RU/VH and our Status dataset (e.g., a lower missing rate), showing that RU/VH scholars—especially women—are more likely to have status appointments in our discipline.
Source: Alter et al. Reference Alter, Jean, Schraudenbach and Rozier2020, tenure-line faculty only.
Lower numbers = less likely to be missing. Gender differences are not statistically significant.
Since not all of our status indicators are the same, we consider where and how women are attaining status. We still exclude section leadership, and we focus on tenure-line RU/VH individuals with some level of status (which is 50.9% of our Status dataset when section leadership is excluded). We find that women are more likely to reach Committee or Officer positions as their highest level of status in comparison to men, while nearly twice as many men reach the Leader level compared to women. The final row of table 4 highlights the differences in highest status for each gender—for example 57.2% of male faculty and 66.3% of female faculty reached the Committee level but no higher, leading to a difference of –9.1%. Even though RU/VH women faculty are less likely overall to be missing from our status dataset (table 3), they are over-represented at the Committee level, and more likely to be missing from the Leader/Honor level. Online appendix 4 expands on this analysis looking at larger pools of faculty and adding section leadership. The numbers vary, but the overall finding holds.
Source: Alter et al. Reference Alter, Jean, Schraudenbach and Rozier2020, section leadership excluded
*** p<0.001, ** p<0.01, * p<0.05
Focusing on Section Leadership and Incorporating Publications
Next we wanted to see if publishing in top American political science journals might be connected to the highest-weight status level that scholars achieved, and we explored the possibility of a tradeoff between service and publication rates. Our ability to test these relationships is limited, however, by the nature of the Teele and Thelen data, which covers fifteen years of publishing in ten top political science journals and by the reality that our own data is not a time series.
Our dependent variable in figure 8 is the section leadership of men and women. We examine the relative prevalence of men and women as section leaders in both the “combined dataset” (a combination of the Status, Teele and Thelen, and Baseline dataset members) and those Baseline dataset members with some coded status or publications. In both the combined and Baseline analysis, we find that women perform more section leadership than their male peers, evidenced by the height difference of the bars and the non-overlap of confidence intervals around these estimates. With this method, we are trying to ensure that we don’t conflate career timing (the possibility that women are a larger share of early-career individuals) with our gender findings. The statistical analysis controls for a scholar’s gender, the number of publications in the Teele-Thelen dataset, and highest-weight status reached. This analysis reveals that if we had two scholars with the same level of status, if the scholar were a female they would contribute roughly 39% more section leadership than a comparable representative male scholar, on average (0.195 to 0.140). When we focus only on faculty at U.S. PhD-granting institutions (Baseline members) we again find that on average a female scholar is likely to perform 47% more section leadership than a comparable male scholar (0.201 to 0.137). Both of these differences are statistically significant. This is another indication that compared to men, section leadership is a key way in which women are leaning in.
Standard errors in parentheses; *** p < 0.001, ** p < 0.01, * p < 0.05
Baseline data analysis is restricted to tenure-track individuals within the Baseline dataset who also have status and journal data (number of names).
The analysis supporting figure 8 also reveals that although scholars with higher-weighted status are also tapped to be section officers, the most likely providers of section leadership are scholars within the status-weight level 1 (Committee level) of leadership. Meanwhile, we found a small negative relationship between publishing and section leadership: it may be that scholars face a trade-off between section leadership and publishing in the journals that Teele and Thelen surveyed. Footnote 10 Online appendix 4 provides additional statistical analysis supporting these findings.
So far, we have demonstrated that both male and female faculty at U.S. PhD-granting institutions are equally likely to appear in our Status dataset, yet female scholars are more likely to achieve a Committee or Officer level of status, not a Leader/Honor level of status. We also demonstrated that compared to their male counterparts, providing section leadership is a key way that women lean in.
Accrued Status: Who Is the Most Rewarded?
Finally, we wanted to know which political scientists garnered the largest amounts of status. We generated an “accrued status” by summing the weighted total status points. Here we report accrued rates omitting section leadership. Of the faculty present in the Status dataset (N=2648 names), the top status earners (those with 4 or more status points (N=396) comprise the top 15% of status-point earners; women comprise 28.5% of this group. Figure 9 breaks down by gender the top status earners, creating four brackets. Given our baseline of 28% female tenured faculty and 23.3% female full professors, our accrued status metric is relatively gender representative. The extent of women’s representation in peak status institutions is noteworthy in comparison to citation data. Considering the entire Kim and Grofman (hereafter shown as KG) dataset, thus including emeritus faculty, the top 400 most cited scholars comprise 8.3% of the KG dataset, and women comprise 14.5% of this top (refer to online appendix 1 for more on the gendered nature of the KG citation data).
We report accrued status for the top 396 individual faculty by name in online appendix 5, listing the accrued status minus section data, then noting section leadership and (where available) the KG citation information for both active and emeritus faculty. Appendix 5 also reports by name KG top 400 active faculty who do not have 4 or more accrued status points. Instead of reporting the names here, figure 10 considers the relationship between citations and cumulative status. One-third of our top status earners (N=133) are omitted from the KG dataset, presumably because they are not based at one of the PhD-granting institutions that KG coded, or because they are at a policy-school or deceased. Of the two-thirds of top accrued status earners with KG data (N= 263), 78 are women (29.7%) and 185 are men (70.3%). Figure 10 shows that women are clustered at lower accrued status and lower citation levels, while a number of men with equal levels of accrued status receive higher citation counts. Some faculty may have accrued status in institutions we did not code, but the clustering difference suggests that men and women accrue status in different ways.
These differences can be individually consequential. Tracing the career mobility of political science faculty cohorts, Kim and Grofman find that faculty with higher citation counts are more likely to advance to full professor and to be at or stay at highly ranked departments, and that “much of the gender bias in academia (i.e. tenure, promotions, etc.) stems from the gender bias in citation counts” (Kim and Grofman Reference Kim and Grofman2019b, 689).
Does Accountability Improve Gender Representation?
We wondered whether oversight, and thus accountability in appointment processes, might generate more representative gender balances. We also wondered whether the nomination process might affect membership, expecting that selection processes that drew on current members to nominate new members (a decentralized process) might reinforce the exclusionary nature of social networks. We explored this idea to the extent that our data allows, categorizing our status positions based on whether selection occurs in the form of accountable and open nomination processes versus unaccountable and member-driven selection processes. The categorization is based on conversations with APSA staff, editors of journals, and members of honor societies. As a bonus, we compare active tenure-line faculty in PhD-granting political science departments (our Baseline dataset) with KG’s Top 400 most cited active faculty. We are assessing institutional accountability, thus whether there is a larger institution that could—in principle—scrutinize gender balances in decision outcomes.
Table 5 reports our categorizations and findings. We find that more monitored/centralized processes typically have higher proportions of women compared to their more ad hoc/decentralized counterparts. We also find that where appointment processes rely on social networks and lack accountability, gender distributions track weighted status levels. In other words, high weighted-status positions underrepresent women scholars, whereas lower weighted-status positions overrepresent women scholars. Most of the findings are not statistically significant, yet overall, we see that accountability leads to higher numbers of women in status positions.
Source: Alter et al. 2020
Asterisks denote statistical significance: *** p<0.001, ** p<0.01, * p<0.5.
Note: We count individuals who have served in each position. 1APSR, POP, AJPS, POQ, PA 2IS, APR, WP, PT, JOP, JCR, IO, CPS, CP, PB
Overall, our findings indicate two different forces generating gender disparities in status positions. For more honorific appointments, a lack of accountability produces an underrepresentation of women. But for status appointments that involve more service-heavy leadership, less visible selection processes overrepresent women, either because selectors are seeking parity or because men refuse nominations for positions that might involve greater service. So we can say that accountability contributes to greater gender representation. It may even contribute to greater recognition of scholars with diverse interests and backgrounds, scholars based at non-PhD or non-American institutions, and scholars who perhaps earn fewer citations because they are women or they are not as entrenched within mainstream scholarly debates.
Conclusions: Who Decides Who Is Notable?
The question of who decides which political scientists have status is not straightforward. In presenting this analysis, it has become apparent that there is very little consensus about what connotes and constitutes individual status. The first part of our paper explained that early status studies were criticized for relying on popularity-contest surveys, and that the search for more “objective” measures produced indicators such as citations, prizes, and publications in top journals, which reinforce male status over female status. If Cole and Singer’s kick-reaction thesis is correct, providing top recognition in ways that favor men can boost male productivity while depressing female productivity. If the profession and discipline as a whole is structured in a gendered way, then how should female scholars respond to Sheryl Sandberg’s suggestion that they lean in to build their status?
Our Status dataset includes the most prestigious institutions within our discipline. Scholarly chops still matter for these positions, but a wider group of political scientists are making selection decisions and there is greater room for intellectual diversity. Selected leaders then enter rooms where leadership networking occurs and discipline-shaping decisions are made. Our finding that women provide significantly more section and mid-level officer leadership, however, suggests that women are doing a greater proportion of caretaking roles for the discipline of political science. This finding mirrors Guarino and Borden’s finding (2017) based on an examination of internal service (e.g., service within home institutions) where women are also caretakers of the “academic family.” Yet we also came to recognize that service-heavy leadership may be the best and perhaps only way for scholars in new or under-valued research areas to circumvent or change status hierarchies in the profession. Footnote 11
We now seek answers to the three questions identified in the introduction:
1. Do institutions within the discipline of political science—including departments, APSA, editorial boards, and academic honor societies—reflect or remedy gender disparities that exist in many forms of recognition, including appointments to top leadership and citations?
Our sad baseline—drawn from the percent of female tenured faculty at US PhD-granting institutions—sets a pretty low bar for assessing whether or not a status institution is gender skewed. Only a finding of less than 28% female presence suggests gender bias; meanwhile where women comprise more than 30% of a status category, men may be the under-represented category.
In aggregate, the peak institutions we study reflect gender balances in our profession, and this is arguably a remedy for older forms of status recognition. Of the named faculty in our Status dataset, 31.3% are women. Overall, 32% of editorial boards and journal leadership are female. Combining all forms of leadership in our largest professional associations—APSA and ISA—40.4% of leaders are women. When one consults the list of top status earners (in online appendix 5), the highest accrued-status faculty are both intellectual and institutional leaders in our profession.
The less hopeful side of the aggregated findings is that the path to accruing status is different for men and women. Breaking down and weighting the different types of status positions, we find once again that women are disproportionately congregated in categories with relatively low-status/high-service. Women were present at higher than expected levels as section officers, within editorial boards, on the council of academic associations, and in mid-level leadership positions. Women also had, on balance, more observations in our Status dataset, suggesting overall that female scholars provide a disproportionate share of leadership and service within the discipline. Meanwhile men were present in higher than expected levels in the honor categories, as plenary speakers at annual meetings, as APSA and ISA president, and as top citation earners.
Also, when institutions strive for symbolic representation that is—at its bottom—a form of service parity, women may be over serviced. Thus, leaders within institutions that seek to appear more gender balanced by filling mid-level appointments with women, and editorial boards such as Comparative Political Studies, American Political Science Review, International Organization, Political Behavior, and Political Theory (refer to online appendix 3) may end up over-servicing women faculty. Section officers may be another place where a misguided goal of gender parity over-taxes women.
To the extent that top honors are influenced by social-networking and highly gendered citation measures, women may have little alternative than to build notability via service-heavy leadership. It is surely true that untenured faculty are ill-advised to assume a leadership position within a section. Yet it is far from clear that tenured faculty should avoid service-heavy leadership altogether. Changing institutions from within may be the best way to circumvent and change existing status hierarchies.
2. Are institutions with centralized and accountable appointment mechanisms less gender skewed compared to more ad hoc selection processes where implicit bias may go unchecked?
Our best sense is that accountability matters, but accountability can cut two ways. Academic departments and leadership in academic associations (president, vice-president and council levels) were less gender skewed compared to gender balances in decentralized/less accountable appointment processes, with the important exception of section leadership, where women are overrepresented. It is also true that some leaders pay attention to gender issues even if the institution itself is not, as a formal matter, accountable to oversight bodies. And the other side of the accountability coin may be an overcompensation where lower-level leadership is used to create gender representation, so that women end up doing a disproportionate share of service-heavy/lower-status leadership.
Overall, however, we must recognize that we are mostly assessing a very weak form of accountability. Our study focused on institutional accountability. Dobbin and Kalev find that institutional accountability mechanisms are relatively ineffective, and at times even counter-productive, in promoting diversity. Instead, social accountability in the form of providing information regarding diversity outcomes encourages managers to follow procedures and to generate defensible diversity outcomes (Dobbin and Kalev Reference Dobbin and Kalev2016). Only recently have certain institutions in political science started to create a more public form of social accountability. Footnote 12 The focus on gender in the journals is an example of social accountability. Our over-time analysis of editorial boards, including a comparison to 2017 (reported in online appendix 3), suggests that the recent focus on gender in the journals has generated gender introspection that cuts across the accountability divide, and it has led to overcompensation in the number of women on some editorial boards. The final section of our conclusion considers how we might increase social accountability more broadly.
3. Does leaning in help? Does the effort of women to publish and to claim a seat at leadership tables increase the likelihood that higher-level status positions will follow?
Our lean-in indicators include appointments to gateway service-heavy status positions (editorial boards, section leadership, APSA/ISA councils) and the publication findings of the Teele and Thelen study. When we focus on the number of observations (how many status roles a person serves) and when we include APSA/ISA section data, we find that compared to men, women are leaning in more, but mostly by providing more service-heavy leadership. This type of leaning in arguably contributes to professional advancement in that building social networks and being asked to assume visible leadership in peak institutions is important for tenure and promotion. And section leadership may build subfield status by creating prizes and membership that raises the profile for new or marginalized areas of research. Footnote 13 But leading and generating sub-field status does not necessarily contribute to greater individual status. Meanwhile, highly cited scholars may be given a pass when it comes to service-heavy leadership.
Women are gaining: when we look at status representation in 2000 versus 2016, it is clear that the percent of women in the top ranked status positions (Leader/Honor) has increased (table 2), as has the percent of full professor women. Assessing the gender of scholars who accrued the most status, there may be no inherent trade-off between providing service-heavy leadership and being a top scholar (refer to online appendix 5). But the path to recognition as female scholars seems to involve relatively more service (and, one might add, relatively fewer citations). Even among our list of the top 396 status-earners, women provide more lower-status section leadership compared to men: 41.6% of the women (N= 47) and 24% of the men (N=68) contributed some level of section leadership. When we consider the gender composition of the Honor and Leader categories (table 4), it is clear that on average lower-level leaning in does not necessarily translate to recognition in the form of appointments to the highest Honor and Leadership categories. While we cannot know what would have happened without these leaning in efforts, it is not clear that leaning in translates to rising to the highest levels of status.
Where Do We Go from Here?
If our discipline is content with a finding that status institutions in aggregate reflect the gender balance of 20% to 30% of female political scientists then this study presents good news. We, however, are not happy with the finding that women are concentrated in lower-status and service heavy positions. Our research provides hints of a pathway forward.
Given that faculty at RU/VH institutions are more likely to gain status appointments (and, according to Hesli and Lee Reference Hesli and Lee.2011, 398, they also publish more articles), our study suggests that the best way to promote gender equity may be to increase the percentage of female faculty at RU/VH institutions. We should also create more social accountability, highlighting which institutions are failing to reflect gender balances in our discipline.
One lesson is that striving for parity in leadership of peak institutions in our discipline may in fact lead to over-servicing women, especially if parity is based on having women fill the service-heavy and middle-manager roles. The larger challenge, however, is that women may be more likely to say “yes” and men are more likely to say “no” to service-heavy requests. Given that women are less likely to gain the sorts of accommodations that ensure that service does not undermine the productive capacity of the scholar (Mitchell and Hesli Reference Mitchell and Hesli2013), over-servicing women—especially when combined with negative kicks—can impact female productivity. The clear solution is that we need to set expectations that encourage all political scientists to take up their share of caretaker positions in our profession. Until women comprise 50% of the professoriate, our expectation should be a 70/30 gender benchmark: men should be fulfilling 70% of Committee and mid-level management roles. Applying this expectation may be easier at the university level, where raises and promotions can be tied to providing service, than the unremunerated discipline-wide level.
A second lesson might concern the need to amplify positive kicks that fuel innovation and scholarly productivity. Our study suggests that we might link recognition to different forms of leaning in, and to various ways of contributing leadership. But more generally, it may be time to review the sorts of honor recognitions that exist in our profession. Maybe we should be looking to those who do the most of what we as a discipline want—be it policy-relevant research, writing for the public, building new fields and modes of inquiry, generating new insight, or leading (however we define it)—when we make decisions about top prizes and accolades. Maybe we should consider citation counts less as an indication of status, but instead as identifying a pool of scholars who have benefited greatly and therefore should be asked to provide more service-heavy/lower-status leadership.
Third, we need to remedy gender bias where it exists, as failing to do so creates negative kicks. Greater social accountability might improve certain metrics, such as citations. It may be time for journals to publish the gender balances in citations, and—where necessary—require that contributors improve balances. Footnote 14 But overall, deemphasizing gendered metrics and creating meaningful positive kicks could stimulate scholars’ productive contributions, helping us capture the full creativity of a larger share of our profession. This correction may also remedy the constant complaint that political science focuses on and rewards scholarship based on discipline-specific values—such as method or certain topics–that are divorced from the issues of the day and the lives of ordinary people.
A final thought is that political scientists can individually and collectively affect these outcomes by creating more social accountability. The problem with the lean-in thesis is that it puts the onus on women to do more. We can ask men to do more service and we can pressure our peak institutions to report gender balances in prizes, publications, and cited publications. Journals and universities could publish internal reviews of gender balances in departments and journals, and journals could publish citation balances. Notable scholars in our field can follow the lead of Francis Collins, the director of the National Institutions of Health who publicly refused to participate in all-male panels, Footnote 15 and they can lead by documenting and promoting gender balance in their syllabi, public presentations, and publications, and by contributing to lower-status service roles. Indeed maybe we should all regularly report gender balances of syllabi, citations within articles, and speaker series. Finally, we can choose to reward those who lean in and provide leadership by letting those who lead garner prestige invitations, honor recognition, and top leadership roles. Greater recognition of all forms of leadership may lead to overrepresentation of women at the highest levels of our profession, a reality that just might encourage more men to demonstrate across their career that top scholarship and high levels of leadership are mutually compatible objectives.
Appendix 1. Description of Data Sets
Appendix 2. APSA and Baseline Datasets Compared
Appendix 3. Gender in the Journals Deeper Analysis
Appendix 4. Additional Statistical Analysis
Appendix 5. Top Cumulative Status Earners
To view supplementary material for this article, please visit https://doi.org/10.1017/S1537592719004985