Hostname: page-component-7c8c6479df-995ml Total loading time: 0 Render date: 2024-03-28T15:01:36.042Z Has data issue: false hasContentIssue false

Toward Better Hiring Practices

Published online by Cambridge University Press:  29 April 2021

Elizabeth Carlson
Affiliation:
North Dakota State University
Christopher Zorn
Affiliation:
Pennsylvania State University
Rights & Permissions [Opens in a new window]

Abstract

Type
Strategies for How Men Can Advance Gender Equity in Political Science
Copyright
© The Author(s), 2021. Published by Cambridge University Press on behalf of the American Political Science Association

Introduction

Observers of gender dynamics in the academy have long characterized academic careers as a “leaky pipeline,” which refers to the tendency for women to occupy a steadily decreasing proportion of academic positions as the rank and status of those positions increase. Among the many loci of such leaks, implicit and explicit biases against women have been shown to affect the hiring process across the entire range of STEM fields (Moss-Racusin et al. Reference Moss-Racusin, Dovidio, Brescoll, Graham and Handelsman2012; Storage et al. Reference Storage, Charlesworth, Banaji and Cimpian2020), including political science.

This article describes the impacts of several hiring practices that offer the potential for reducing gender-related biases in that process. Our description takes the form of a case study, focusing a faculty search at a Carnegie “Doctoral—Very High Research Activity” (“R1”) university in a political science subfield that has been and remains overwhelmingly male dominated: political methodology. The innovations include establishing clear ex ante criteria for evaluating applicants, emphasizing “fit” to the position as advertised and postponing reading letters of recommendation until candidates were ranked in a “long list” based on other evaluation criteria. Although many of these strategies increasingly are being adopted as best practices, our case study provides evidence of the immediate effects of these strategies on gender balance, both on the search in question and (briefly) in subsequent searches. Our experience suggests that such procedural changes offer the potential to increase gender diversity in the hiring process. Moreover, in most instances, these innovations are simple and largely costless to adopt.

Our experience suggests that such procedural changes offer the potential to increase gender diversity in the hiring process. Moreover, in most instances, these innovations are simple and largely costless to adopt.

The Search

In the two decades since political methodology was famously (if hilariously) described as a “welcoming discipline” (Beck Reference Beck2000), numerous studies have documented the pervasive, persistent gender imbalance in scholars seeking, making, and continuing careers in political methodology (Breuning and Sanders Reference Sanders2007; Reference Dion, Sumner and MitchellDion, Sumner, and Mitchell Reference Dion, Sumner and Mitchell2018; Roberts Reference Roberts2018; Shames and Wise Reference Shames and Wise2017; Teele and Thelen Reference Teele and Thelen2017). As recently as 2018, a report by the Diversity Committee of the American Political Science Association (APSA) Society for Political Methodology noted that “the political methodology field faces severe diversity challenges” (Hidalgo et al. Reference Hidalgo, Linn, Roberts, Sinclair and Titiunik2018, 7).

The political science department at Pennsylvania State University recently received authorization to conduct an open-rank search for a tenure-track professor with a specialization in quantitative methodology.Footnote 1 The authors were appointed to the five-member search committee, with Zorn—a senior male faculty member—acting as committee chair. Carlson, then a junior faculty member in the department, was the committee’s only female member.

At the time of the search, the department’s climate committee had created a draft memorandum outlining a series of best practices for departmental search committees, in keeping with practices being increasingly adopted by R1 universities. Those practices included establishing clear criteria for a successful applicant and then independently rating candidates on each of those criteria. This practice is intended to prevent unstated, subjective—and potentially biased—criteria from eliminating objectively qualified candidates. In adopting these recommendations, the search committee agreed on the following four criteria on which candidates would be scored:

  • research and teaching that fit the specific needs identified in the advertisement

  • a high-quality publication and grant record (or the promise thereof)

  • a compelling political science research agenda

  • record (or intention) of commitment or contribution to improving departmental climate and diversity in political science

Members of the search committee were instructed to review each candidate’s file and to rate them on each of these four criteria using an 11-point scale from 0 (worst possible) to 10 (best possible), with 3 being the minimum “acceptable” rating.Footnote 2

In preliminary meetings of the search committee, Carlson noted the demonstrated tendency of women to apply for positions for which they meet all stated criteria, whereas men are more likely to apply broadly, including for positions for which they do not meet one or more of the qualifications stated in the advertisement (Ceci et al. Reference Ceci, Ginther, Kahn and Williams2014). She suggested that to combat this, “fit” be interpreted strictly and given particular weight in the rubric.Footnote 3 By emphasizing fit, the committee thus screened out what historically have been called “best-athlete” candidates: individuals with impressive credentials who nonetheless were poor matches with a specific position. This category of applicants is likely to be disproportionately male and may exclude a number of equally strong female “athletes” who did not apply for the job because they did not believe they were a good fit for the advertised position.Footnote 4 Noting these dynamics, the committee agreed to weight applicants’ scores on the four criteria, with the fit and publication criteria receiving greater weight (0.4 each) and the agenda and climate criteria weighted less (0.1 each).Footnote 5

Carlson also noted the tendency for letters of recommendation to display bias (conscious or implicit) against female job applicants, typically without the awareness of the applicant,Footnote 6 and she suggested that letters not be considered in the hiring process. However, recognizing that letters of recommendation also provide information that can benefit a candidate, the committee instead decided to read the letters—but only after a long list of eight to 10 candidates was generated based on other application materials. This meant that the materials considered by the search committee at the initial evaluation stage—including cover letters, curricula vitae, examples of published and unpublished research, and teaching materials—consisted entirely of materials over which the applicant had complete control.Footnote 7

The committee received and reviewed a total of 53 complete application files for the position. Of those applicants, 15 (28.3%) were female. This percentage was substantially lower than the percentage of women holding tenure-track positions in political science but higher than (for example) the share of female members of the Society for Political Methodology.Footnote 8 Committee members scored each of the 53 candidates on each of the four criteria; the committee chair then gathered and analyzed those assessments before the decision-making meeting. Figure 1 summarizes results of the committee’s candidate-rating process.Footnote 9 The left-hand panel reports the (unweighted, standardized) mean ratings on each of the four search criteria for male (in blue) and female (in orange) candidates, along with t-statistics for differences of means between male and female applicants. As expected, female candidates generally scored higher in terms of fit to the position. Indeed, categorized dichotomously into “good” (6) and “poor (<6) fits, half of those deemed to be a good fit were women, whereas more than three quarters of those deemed to be a poor fit were men. Therefore, weighting fit more heavily increased the average scores of women in the applicant pool. Conversely, male candidates scored higher, on average, on the criterion related to publications and grants, a result consistent with recent research on publication bias in the discipline (Teele and Thelen Reference Teele and Thelen2017). Although there were no substantial differences between men and women regarding their research agenda, female candidates scored notably higher, on average, with respect to their contribution to diversity and departmental climate.

Figure 1 Standardized Score Means by Gender and Candidate-Criteria Biplot

The left panel plots the means of the unweighted standardized scores for the candidates, by gender. The right panel is the biplot of candidates and search criteria. In both plots, symbols for female candidates are in orange, male candidates are in blue. See the main text for details.

The right-hand panel of figure 1 is a biplot summarizing the data regarding the candidates’ ratings on the search criteria. The horizontal axis—representing the first extracted principal component—shows that the fit and research-agenda criteria load similarly to one another, with publications also loading strongly on that axis. The criterion related to climate loads strongly on the vertical axis and is mostly orthogonal to the other three criteria. In addition, although both male and female candidates are arrayed across a range of values on both axes, female candidates are notably absent from the lower range on the component mapping most strongly onto fit/agenda while also consistently loading strongly on the axis relating to climate. This pattern reinforces the general finding that placing greater emphasis on fit to the position works to mitigate at least some of the biases against female candidates and to remove any advantage for male “best athletes.” Upweighting climate, which we did not do, might have increased further the number of women on the short list. These results provide evidence contrary to Stacy et al. (Reference Stacy, Goulden, Frasch and Broughton2018), who found that searches using weighted rubrics are not more effective in hiring women.

Following this analysis, the committee chair summed scores from the (standardized, weighted) committee ratings; those summary scores then provided the basis for the committee’s initial winnowing of the candidate pool. Using the weighted-sum scores, two of the top four candidates and four of the top eight candidates were female. Across all candidates, the mean (standardized, weighted) sum score for male candidates was -0.05; for female candidates, the average was 0.12 (t=1.1). After review and discussion, the search committee created a long list of 10 candidates; it then adjourned to read the letters of recommendation for those 10 candidates. In subsequent discussion, committee members noted a number of instances in which bias appeared to shape letters and indicated that, in a few examples, if those letters had been read before the candidate’s own research, they might have led to committee members lowering their evaluation of the candidate’s promise. The committee then reconvened and agreed to recommend that the department interview five candidates, three of whom were female; the department agreed and proceeded with the interview process.

In retrospect, several aspects of this search process are notable. Primary among them was the outcome: in a subfield of political science known for the underrepresentation of women, the search processes led to a high degree of gender balance in the candidate long list and interview pool. Moreover, the department’s experience with adopting similar procedures in subsequent searches has led to similar results. A search for a nearly identical position the following academic year resulted in an interview pool that was two thirds female. In the ensuing years, use of these procedures for searches in American politics and international relations has resulted in the department interviewing and hiring significant numbers of female candidates, as well as a number of faculty from historically underrepresented groups. Notably, in every such search, the committee chair was a male faculty member who applied these protocols to overcome structural bias, thereby increasing the share of women being interviewed and hired. Although it is likely that these documented changes do not explain the successes, we believe that they contributed to them.

Conclusions and Implications

In the introduction to a recent symposium in The Washington Post, Dionne (Reference Dionne2019) reviewed the lamentable state of women in academic political science. She also noted that “(T)he political science ecosystem may be on the verge of a big shift” because increasing awareness of gender disparities has begun (it is hoped) to lead to meaningful changes in the institutions and practices of our discipline. Our description of a series of innovations made to the hiring process—designed to counteract institutional and behavioral dynamics that work to decrease the representation of women in the discipline—is an example of these changes. Our review of that process suggests that a few relatively simple changes can contribute to greater gender balance in the field. Moreover, these changes are relatively simple and costless to adopt.

These are practices that can (and, we think, should) be implemented by anyone conducting a search in political science. However, given the historical and (resulting) demographic composition of most political science departments, it is likely that most department chairs/heads and most search committee chairs are male (Mitchell and Hesli Reference Mitchell and Hesli2013). Of course, this is particularly likely to be the case in fields that historically have been male dominated (Charlesworth and Banaji Reference Charlesworth and Banaji2019). Within political science, this includes the subfields of quantitative methodology, formal theory, normative political theory, and international relations. Indeed, the Society for Political Methodology’s diversity report recently noted that “the majority of the positions of (formal and informal) power in our field are occupied by non-minority men” (Hidalgo et al. Reference Hidalgo, Linn, Roberts, Sinclair and Titiunik2018, 12). To the extent that such changes are implemented by primarily male faculty and administrators, their effectiveness offers the potential for multiplier effects because higher numbers of female faculty in turn may be empowered to assume these roles.

Data Availability Statement

Replication materials are available on Harvard Dataverse at https://doi.org/10.7910/DVN/21CU9M.

Supplementary Materials

To view supplementary material for this article, please visit http://dx.doi.org/10.1017/S1049096521000196.

Footnotes

1. The full advertisement for the position, as it appeared on APSA’s eJobs portal, is reprinted in the online appendix.

2. At other universities, evaluators are asked to point to the source for their rating. Although we support this practice and encourage others to apply it, we did not include it as a formal step in this search.

3. We want to emphasize that “fit” in this case refers to the match between a candidate’s expertise and the stated needs in the job advertisement rather than a vague or impressionistic fit with department culture. This latter type of “fit” can reduce diversity on the short list by penalizing those who are different in some way from current department members.

4. When departments are interested in hiring a “best athlete,” this can be accomplished best by advertising a position with an open specialty.

5. There are two important aspects of our chosen weighting scheme. First, the weights were agreed to by the search committee before the examination and evaluation of candidates’ files. Second, they reflected the consensus of the search committee concerning each criterion’s relative importance, considering the nature of the institution and the position. We expect that search committees for other positions, at other institutions, might collectively agree to adopt weighting criteria appropriate for their circumstances.

6. For example, Dutt et al. (Reference Dutt, Pfaff, Bernstein, Dillard and Block2016) showed that female postdoctoral candidates are 50% as likely as their male counterparts to receive “excellent” letters of recommendation, and Madera et al. (Reference Madera, Hebl, Dial, Martin and Valian2019) showed that letter writers use less decisive language about female applicants.

7. Other institutions follow a similar strategy by soliciting letters of recommendation only after candidates have been placed on the short list. When that is not feasible, delaying the reading of the letters can serve the same purpose.

8. At the time of the search, women comprised approximately 40% of all political science faculty (Shames and Wise Reference Shames and Wise2017, 814) but less than 20% among members of the Society for Political Methodology (Hidalgo et al. Reference Hidalgo, Linn, Roberts, Sinclair and Titiunik2018, 2), the lowest percentage of all APSA organized sections.

9. For this article, the names of the candidates were changed to random, gender-consistent names using the randomNames package in R Code. De-identified data to reproduce the analyses presented here are available at https://github.com/PrisonRodeo/TBHP-git, and at the Dataverse accompanying this article (Carlson and Zorn Reference Carlson and Zorn2021).

References

REFERENCES

Beck, Nathaniel L. 2000. “Political Methodology: A Welcoming Discipline.” Journal of the American Statistical Association 95:651–54.CrossRefGoogle Scholar
Breuning, Marijke, and Sanders, Kathryn. 2007. “Gender and Journal Authorship in Eight Prestigious Political Science Journals.” PS: Political Science & Politics 40 (2): 347–51.Google Scholar
Carlson, Elizabeth, and Zorn, Christopher. 2021. “Replication Data for: Toward Better Hiring Practices.” Harvard Dataverse, DOI:10.7910/DVN/21CU9M. CrossRefGoogle Scholar
Ceci, Stephen J., Ginther, Donna K., Kahn, Shulamit, and Williams, Wendy M.. 2014. “Women in Academic Science: A Changing Landscape.” Psychological Science in the Public Interest 15 (3): 75141.CrossRefGoogle ScholarPubMed
Charlesworth, Tessa E. S., and Banaji, Mahzarin R.. 2019. “Gender in Science, Technology, Engineering, and Mathematics: Issues, Causes, Solutions.” Journal of Neuroscience 39 (37): 7228–43.CrossRefGoogle ScholarPubMed
Dion, Michelle L., Sumner, Jane L., and Mitchell, Sara McLaughlin. 2018. “Gendered Citation Patterns across Political Science and Social Science Methodology Fields.” Political Analysis 26 (3): 312–27.CrossRefGoogle Scholar
Dionne, Kim Yi. 2019. “There’s a Gender Gap in Political Science: Our Series Examines the Problem—and Looks at Some Solutions.” The Washington Post , August 19.Google Scholar
Dutt, Kuheli, Pfaff, Danielle L., Bernstein, Ariel F., Dillard, Joseph S., and Block, Caryn J.. 2016. “Gender Differences in Recommendation Letters for Postdoctoral Fellowships in Geoscience.” Nature Geoscience 9 (11): 805808.CrossRefGoogle Scholar
Hidalgo, F. Daniel, Linn, Suzanna, Roberts, Margaret, Sinclair, Betsy, and Titiunik, Rocío. 2018. “Report on Diversity and Inclusion in the Society for Political Methodology.” Society for Political Methodology. Technical report, February 21.Google Scholar
Madera, Juan M., Hebl, Michelle R., Dial, Heather, Martin, Randi, and Valian, Virginia. 2019. “Raising Doubt in Letters of Recommendation for Academia: Gender Differences and Their Impact.” Journal of Business and Psychology 34 (3): 287303.CrossRefGoogle Scholar
Mitchell, Sara McLaughlin, and Hesli, Vicki L.. 2013. “Women Don’t Ask? Women Don’t Say No? Bargaining and Service in the Political Science Profession.” PS: Political Science & Politics 46 (2): 355–69.Google Scholar
Moss-Racusin, Corinne A., Dovidio, John F., Brescoll, Victoria L., Graham, Mark J., and Handelsman, Jo. 2012. “Science Faculty’s Subtle Gender Biases Favor Male Students.” Proceedings of the National Academy of Sciences 109 (41): 16474–79.CrossRefGoogle ScholarPubMed
Roberts, Margaret E. 2018. “What Is Political Methodology? PS: Political Science & Politics 51 (3): 597601.Google Scholar
Shames, Shauna L., and Wise, Tess. 2017. “Gender, Diversity, and Methods in Political Science: A Theory of Selection and Survival Biases.” PS: Political Science & Politics 50 (3): 811–23.Google Scholar
Stacy, Angelica, Goulden, Marc, Frasch, Karie, and Broughton, Janet. 2018. “Searching for Diverse Faculty: Data-Driven Recommendations.” University of California, Berkeley: Technical report. https://ofew.berkeley.edu/equity/uc-berkeley-data.Google Scholar
Storage, Daniel, Charlesworth, Tessa E. S., Banaji, Mahzarin R., and Cimpian, Andrei. 2020. “Adults and Children Implicitly Associate Brilliance with Men More Than Women.” Journal of Experimental Social Psychology 90:104020.CrossRefGoogle Scholar
Teele, Dawn Langan, and Thelen, Kathleen. 2017. “Gender in the Journals: Publication Patterns in Political Science.” PS: Political Science & Politics 50 (2): 433–47.Google Scholar
Figure 0

Figure 1 Standardized Score Means by Gender and Candidate-Criteria BiplotThe left panel plots the means of the unweighted standardized scores for the candidates, by gender. The right panel is the biplot of candidates and search criteria. In both plots, symbols for female candidates are in orange, male candidates are in blue. See the main text for details.

Supplementary material: Link

Carlson and Zorn Dataset

Link
Supplementary material: PDF

Carlson and Zorn supplementary material

Carlson and Zorn supplementary material 1

Download Carlson and Zorn supplementary material(PDF)
PDF 126.8 KB
Supplementary material: Image

Carlson and Zorn supplementary material

Carlson and Zorn supplementary material 2

Download Carlson and Zorn supplementary material(Image)
Image 25.9 MB