Hostname: page-component-76fb5796d-dfsvx Total loading time: 0 Render date: 2024-04-25T11:16:19.545Z Has data issue: false hasContentIssue false
  • English
  • Français

Addressing the so-called validity–diversity trade-off: Exploring the practicalities and legal defensibility of Pareto-optimization for reducing adverse impact within personnel selection

Published online by Cambridge University Press:  28 July 2020

Deborah E. Rupp ,
Q. Chelsea Song  and
Nicole Strah
Deborah E. Rupp*
Affiliation:
George Mason University
Q. Chelsea Song
Affiliation:
Purdue University
Nicole Strah
Affiliation:
Purdue University
*
*Corresponding author. Email: drupp2@gmu.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

It is necessary for personnel selection systems to be effective, fair, and legally appropriate. Sometimes these goals are complementary, whereas other times they conflict (leading to the so-called “validity-diversity dilemma”). In this practice forum, we trace the history and legality of proposed approaches for simultaneously maximizing job performance and diversity through personnel selection, leading to a review of a more recent method, the Pareto-optimization approach. We first describe the method at various levels of complexity and provide guidance (with examples) for implementing the technique in practice. Then, we review the potential points at which the method might be challenged legally and present defenses against those challenges. Finally, we conclude with practical tips for implementing Pareto-optimization within personnel selection.

Keywords

personnel selectionPareto-optimal weightingadverse impactvalidity-diversitydiscriminationrace discrimination
Type
Practice Forum
Information
Industrial and Organizational Psychology , Volume 13 , Issue 2 , June 2020 , pp. 246 - 271
Copyright
© Society for Industrial and Organizational Psychology, Inc. 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

All authors contributed equally to this article.

References

Aiken, J. R., & Hanges, P. J. (2017). The sum of the parts. In Farr, J. L & Tippins, N. T. (Eds.), Handbook of employee selection (2nd ed., pp. 388). New York, NY: Routledge.10.4324/9781315690193-17CrossRefGoogle Scholar
Black Law Enforcement Officers Assoc. v. Akron,. U.S. Dist. LEXIS 30160 (1986).Google Scholar
Bobko, P. (2001). Correlation and regression: Applications for industrial organizational psychology and management. Thousand Oaks, CA: Sage.10.4135/9781412983815CrossRefGoogle Scholar
Bobko, P., & Roth, P. L. (2013). Reviewing, categorizing, and analyzing the literature on Black–White mean differences for predictors of job performance: Verifying some perceptions and updating/correcting others. Personnel Psychology, 66, 91–126.10.1111/peps.12007CrossRefGoogle Scholar
Bobko, P., Roth, P. L., & Buster, M. A. (2007). The usefulness of unit weights in creating composite scores: A literature review, application to content validity, and meta-analysis. Organizational Research Methods, 10, 689–709.10.1177/1094428106294734CrossRefGoogle Scholar
Bobko, P., Roth, P. L., & Potosky, D. (1999). Derivation and implications of a meta-analytic matrix incorporating cognitive ability, alternative predictors, and job performance. Personnel Psychology, 52, 561–589.10.1111/j.1744-6570.1999.tb00172.xCrossRefGoogle Scholar
Bosco, F., Allen, D. G., & Singh, K. (2015). Executive attention: An alternative perspective on general mental ability, performance, and subgroup differences. Personnel Psychology, 68, 859–898.10.1111/peps.12099CrossRefGoogle Scholar
Bureau of Labor Statistics. (2000). Employment and earnings, 46(1), 13–14 (Chart A-4).Google Scholar
Campion, M. A., Outtz, J. L., Zedeck, S., Schmidt, F. L., Kehoe, J. F., Murphy, K. R., & Guion, R. M. (2001). The controversy over score banding in personnel selection: Answers to 10 key questions. Personnel Psychology, 54, 149–185.10.1111/j.1744-6570.2001.tb00090.xCrossRefGoogle Scholar
Cascio, W., Jacobs, R., & Silva, J. (2009). Validity, utility, and adverse impact: Practical implications from 30 years of data. In Outtz, J. (Ed.), Adverse impact: Implications for organizational staffing and high stakes selection (pp. 217–288). New York: Routledge.Google Scholar
Cascio, W. F., & Aguinis, H. (2011). Criteria: Concepts, measurement, and evaluation. In Applied Psychology in Human Resource Management. Harlow, Essex, UK: Pearson Education Limited.Google Scholar
Civil Rights Act of 1964 § 7, 42 U.S.C. § 2000e et seq (1964).Google Scholar
Connecticut v. Teal, 457 US 440. (1982).Google Scholar
Cottrell, J. M., Newman, D. A., & Roisman, G. I. (2015). Explaining the black–white gap in cognitive test scores: Toward a theory of adverse impact. Journal of Applied Psychology, 100, 1713–1736.CrossRefGoogle Scholar
Darr, W. A., & Catano, V. M. (2016). Determining predictor weights in military selection: An application of dominance analysis. Military Psychology, 28, 193–208.10.1037/mil0000107CrossRefGoogle Scholar
Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993).Google Scholar
Das, I., & Dennis, J. E. (1998). Normal-boundary intersection: A new method for generating the Pareto surface in nonlinear multicriteria optimization problems. SIAM Journal on Optimization, 8(3), 631–657.10.1137/S1052623496307510CrossRefGoogle Scholar
De Corte, W. (2014). Predicting the outcomes of single- and multistage selections when applicant pools are small and heterogeneous. Organizational Research Methods, 17, 412–432.10.1177/1094428114537877CrossRefGoogle Scholar
De Corte, W., Lievens, F., & Sackett, P. R. (2007). Combining predictors to achieve optimal trade-offs between selection quality and adverse impact. Journal of Applied Psychology, 92, 1380–1393.10.1037/0021-9010.92.5.1380CrossRefGoogle ScholarPubMed
De Corte, W., Lievens, F., & Sackett, P. R. (2008). Validity and adverse impact potential of predictor composite formation. International Journal of Selection and Assessment, 16, 183–194.CrossRefGoogle Scholar
De Corte, W., Sackett, P., & Lievens, F. (2010). Selecting predictor subsets: Considering validity and adverse impact. International Journal of Selection and Assessment, 18(3), 260–270.10.1111/j.1468-2389.2010.00509.xCrossRefGoogle Scholar
De Corte, W., Sackett, P. R., & Lievens, F. (2011). Designing Pareto-optimal selection systems: Formalizing the decisions required for selection system development. Journal of Applied Psychology, 96, 907–926.CrossRefGoogle ScholarPubMed
De Corte, W., Sackett, P. R., & Lievens, F. (in press). Robustness, sensitivity, and sampling variability of Pareto-optimal selection system solutions to address the quality-diversity trade-off. Organizational Research Methods. https://doi.org/10.1177/1094428118825301Google Scholar
De Soete, B., Lievens, F., & Druart, C. (2012). An update on the diversity-validity dilemma in personnel selection: A review. Psychological Topics, 21, 399–424.Google Scholar
De Soete, B., Lievens, F., & Druart, C. (2013). Strategies for dealing with the diversity-validity dilemma in personnel selection: Where are we and where should we go? Journal of Work and Organizational Psychology, 29(1), 3–12.Google Scholar
Druart, C., & De Corte, W. (2010). Optimizing the efficiency: Adverse impact trade-off in personnel classification decisions. In 25th Annual Conference of the Society for Industrial and Organizational Psychology. Academia Press.Google Scholar
Druart, C., & De Corte, W. (2012a). Designing Pareto-optimal systems for complex selection decisions. Organizational Research Methods, 15, 488–513.10.1177/1094428112440328CrossRefGoogle Scholar
Druart, C., & De Corte, W. (2012b). Computing Pareto-optimal predictor composites for complex selection decisions. International Journal of Selection and Assessment, 20, 385–393.CrossRefGoogle Scholar
Einhorn, H. J., & Hogarth, R. M. (1975). Unit weighting schemes for decision making. Organizational Behavior & Human Performance, 13, 171–192.10.1016/0030-5073(75)90044-6CrossRefGoogle Scholar
Equal Employment Opportunity Commission, Civil Service Commission, Department of Labor & Department of Justice (EEOC). (1978). Uniform guidelines on employee selection procedures. Federal Register, 43, 38290–39315.Google Scholar
Finch, D. M., Edwards, B. D., & Wallace, J. C. (2009). Multistage selection strategies: Simulating the effects on adverse impact and expected performance for various predictor combinations. Journal of Applied Psychology, 94, 318–340.10.1037/a0013775CrossRefGoogle ScholarPubMed
Gatewood, R., Feild, H. S., & Barrick, M. (2015). Human resource selection. Boston, MA: Cengage Learning.Google Scholar
Griggs v. Duke Power Co., 401 U.S. 424 (1971).Google Scholar
Gutman, A., Koppes, L. L., & Vodanovich, S. J. (2012). EEO law and personnel practices. New York, NY: Psychology Press.Google Scholar
Gutman, A., Outtz, J. L., Dunleavy, E. (2017). An updated sampler of legal principles in employment selection. In Farr, J. L. & Tippins, N. T. (Eds.), Handbook of employee selection (2nd ed., pp. 631–658). New York, NY: Routledge.CrossRefGoogle Scholar
Hattrup, K., Rock, J., & Scalia, C. (1997). The effects of varying conceptualizations of job performance on adverse impact, minority hiring, and predicted performance. Journal of Applied Psychology, 82, 656–664.10.1037/0021-9010.82.5.656CrossRefGoogle Scholar
Hayden v. County of Nassau, 180 F.3d 42 (2nd Cir. 1999).Google Scholar
Henle, C. A. (2004). Case review of the legal status of banding. Human Performance, 17, 415–432.10.1207/s15327043hup1704_4CrossRefGoogle Scholar
Kaufmann, P. M. (2009). Protecting raw data and psychological tests from wrongful disclosure: A primer on the law and other persuasive strategies. The Clinical Neuropsychologist, 23, 1130–1159.10.1080/13854040903107809CrossRefGoogle ScholarPubMed
Kaye, D. H. (2001). The dynamics of Daubert: Methodology, conclusions, and fit in statistical and econometric studies. Virginia Law Review, 87, 1933–2018.CrossRefGoogle Scholar
Kehoe, J. F. (2008). Commentary on Pareto-optimality as a rationale for adverse impact reduction: What would organizations do? International Journal of Selection and Assessment, 16, 195–200.CrossRefGoogle Scholar
Köhn, H. F. (2011). A review of multiobjective programming and its application in quantitative psychology. Journal of Mathematical Psychology, 55, 386–396.CrossRefGoogle Scholar
Kraft, D. (1988). A software package for sequential quadratic programming. Forschungsbericht- Deutsche Forschungs- und Versuchsanstalt fur Luft- und Raumfahrt. Retrieved from http://degenerateconic.com/wp-content/uploads/2018/03/DFVLR_FB_88_28.pdf.Google Scholar
Lievens, F. (2015). Diversity in medical school admission: Insights from personnel recruitment and selection. Medical Education, 49, 11–14.CrossRefGoogle Scholar
Lopes, L. G. D., Brito, T. G., Paiva, A. P., Peruchi, R. S., & Balestrassi, P. P. (2016). Robust parameter optimization based on multivariate normal boundary intersection. Computers & Industrial Engineering, 93, 55–66.10.1016/j.cie.2015.12.023CrossRefGoogle Scholar
McKay, P. F., & McDaniel, M. A. (2006). A reexamination of black-white mean differences in work performance: More data, more moderators. Journal of Applied Psychology, 91, 538–554.CrossRefGoogle ScholarPubMed
Meade, A. W., Thompson, I. B., & Schwall, A. R. (2017). Optimizing validity while controlling adverse impact with ant colony optimization. Paper presented at the Annual Conference of the Society for Industrial and Organizational Psychology, Orlando, FL.Google Scholar
Morris, S. (2017). Statistical significance testing in adverse impact analysis. In Morris, S. B. & Dunleavy, E. M. (Eds.), Adverse impact analysis: Understanding data, statistics and risk (pp. 92–112). New York, NY: Routledge.Google Scholar
Motowidlo, S. J., & Kell, H. J. (2012). Job performance. In Weiner, I., Schmitt, N. W., & Highhouse, S. (Eds.), Handbook of psychology, Vol. 12: Industrial and organizational psychology (2nd ed., pp. 82–103). Hoboken, NJ: John Wiley & Sons, Inc.Google Scholar
Murphy, K. R. (2018). The legal context of the management of human resources. Annual Review of Organizational Psychology and Organizational Behavior, 5, 157–182.CrossRefGoogle Scholar
Naves, F. L., de Paula, T. I., Balestrassi, P. P., Braga, W. L. M., Sawhney, R. S., & de Paiva, A. P. (2017). Multivariate normal boundary intersection based on rotated factor scores: A multiobjective optimization method for methyl orange treatment. Journal of Cleaner Production, 143, 413–439.CrossRefGoogle Scholar
Newman, D. A., Jacobs, R. R., & Bartram, D. (2007). Choosing the best method for local validity estimation: Relative accuracy of meta-analysis versus a local study versus Bayes-analysis. Journal of Applied Psychology, 92(5), 1394–1413.10.1037/0021-9010.92.5.1394CrossRefGoogle ScholarPubMed
Newman, D. A., Jones, K. S., Fraley, R. C., Lyon, J. S., & Mullaney, K. M. (2013). Why minority recruiting doesn’t often work, and what can be done about it: Applicant qualifications and the 4-group model of targeted recruiting. In Yu, K. Y. T. & Cable, D. M. (Eds.), The Oxford handbook of recruitment (pp. 492–526). New York, NY: Oxford University Press.Google Scholar
Newman, D. A., & Lyon, J. S. (2009). Recruitment efforts to reduce adverse impact: Targeted recruiting for personality, cognitive ability, and diversity. Journal of Applied Psychology, 94, 298–317.CrossRefGoogle ScholarPubMed
Nimmegeers, P., Telen, D., Logist, F., & Van Impe, J. (2016). Dynamic optimization of biological networks under parametric uncertainty. BMC Systems Biology, 10(1), 86.10.1186/s12918-016-0328-6CrossRefGoogle ScholarPubMed
Oswald, F. L., Dunleavy, E., & Shaw, A. (2017). Measuring practical significance in adverse impact analysis. In Morris, S. B. & Dunleavy, E. M. (Eds.). Adverse impact analysis: Understanding data, statistics and risk (pp. 92–112). New York, NY: Routledge.Google Scholar
Oswald, F. L., Putka, D. J., Ock, J., Lance, C. E., & Vandenberg, R. J. (2014). Weight a minute, what you see in a weighted composite is probably not what you get. In Lance, C. E. & Vandenburg, R. J. (Eds.), More statistical and methodological myths and urban legends: Doctrine, verity and fable in organizational and social sciences (pp. 187–205). London, UK: Routledge.Google Scholar
Outtz, J. L., & Newman, D. A. (2010). A theory of adverse impact. In Adverse Impact (pp. 80–121). New York: Routledge.CrossRefGoogle Scholar
Ployhart, R. E., & Holtz, B. C. (2008). The diversity–validity dilemma: Strategies for reducing racioethnic and sex subgroup differences and adverse impact in selection. Personnel Psychology, 61, 153–172.10.1111/j.1744-6570.2008.00109.xCrossRefGoogle Scholar
Porter, M. G. (2016). An application of Pareto-optimality to public safety selection data: Assessing the feasibility of optimal composite weighting. Illinois Institute of Technology.Google Scholar
Potosky, D., Bobko, P., & Roth, P. L. (2005). Forming composites of cognitive ability and alternative measures to predict job performance and reduce adverse impact: Corrected estimates and realistic expectations. International Journal of Selection and Assessment, 13(4), 304–315.10.1111/j.1468-2389.2005.00327.xCrossRefGoogle Scholar
Potosky, D., Bobko, P., & Roth, P. L. (2008). Some comments on Pareto thinking, test validity, and adverse impact: When ‘and’ is optimal and ‘or’ is a trade-off. International Journal of Selection and Assessment, 16, 201–205.10.1111/j.1468-2389.2008.00425.xCrossRefGoogle Scholar
Pyburn, K. M Jr., Ployhart, R. E., & Kravitz, D. A. (2008). The diversity–validity dilemma: Overview and legal context. Personnel Psychology, 61, 143–151.10.1111/j.1744-6570.2008.00108.xCrossRefGoogle Scholar
Reynolds v. Ala. DOT, 295 F. Supp. 2d 1298 (2003).Google Scholar
Ricci v. DeStefano, 557 U.S. 557 (2009).Google Scholar
Rickes, S. (2018). Traditional and nontraditional predictors of college FYGPA and racial/ethnic subgroup differences. (Doctoral dissertation, Alliant International University).Google Scholar
Roth, P. L., BeVier, C. A., Bobko, P., Switzer, F. S III., & Tyler, P. (2001). Ethnic group differences in cognitive ability in employment and educational settings: A meta-analysis. Personnel Psychology, 54, 297–330.CrossRefGoogle Scholar
Roth, P. L., Switzer, F. S III., Van Iddekinge, C. H., & Oh, I. S. (2011). Toward better meta-analytic matrices: How input values can affect research conclusions in human resource management simulations. Personnel Psychology, 64, 899–935.CrossRefGoogle Scholar
Russell, T. L., Ford, L., & Ramsberger, P. (2014). Thoughts on the future of military enlisted selection and classification (HumRRO Technical Report No. 053). Alexandria, VA: Human Resources Research Organization (HumRRO).Google Scholar
Ryan, A. M., & Ployhart, R. E. (2014). A century of selection. Annual Review of Psychology, 65, 693–717.10.1146/annurev-psych-010213-115134CrossRefGoogle Scholar
Sackett, P. R., Dahlke, J. A., Shewach, O. R., & Kuncel, N. R. (2017). Effects of predictor weighting methods on incremental validity. Journal of Applied Psychology, 102(10), 1421–1434.CrossRefGoogle ScholarPubMed
Sackett, P. R., De Corte, W., & Lievens, F. (2008). Pareto-optimal predictor composite formation: A complementary approach to alleviating the selection quality/adverse impact dilemma. International Journal of Selection and Assessment, 16, 206–209.10.1111/j.1468-2389.2008.00426.xCrossRefGoogle Scholar
Sackett, P. R., De Corte, W., & Lievens, F. (2010). Decision aids for addressing the validity-adverse impact tradeoff: Implications for organizational staffing and high stakes selection. In Outtz, J. (Ed.), Adverse impact: Implications for organizational staffing and high stakes selection (pp. 459–478). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
Sackett, P. R., & Ellingson, J. E. (1997). The effects of forming multi-predictor composites on group differences and adverse impact. Personnel Psychology, 50, 707–721.CrossRefGoogle Scholar
Sackett, P. R., & Lievens, F. (2008). Personnel selection. Annual Review of Psychology, 59, 419–450.CrossRefGoogle ScholarPubMed
Schmitt, N. (2014). Personality and cognitive ability as predictors of effective performance at work. Annual Review Organizational Psychology and Organizational Behavior, 1, 45–65.CrossRefGoogle Scholar
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262–274.CrossRefGoogle Scholar
Society for Industrial and Organizational Psychology (SIOP). (2018). Principles for the validation and use of personnel selection procedures (5th ed.). Industrial and Organizational Psychology, 11(S1), 1–97.Google Scholar
Song, Q. C. (2018). Diversity shrinkage of Pareto-optimal solutions in hiring practice: Simulation, shrinkage formula, and a regularization technique (Doctoral dissertation, University of Illinois at Urbana-Champaign, Urbana, IL).Google Scholar
Song, Q. C., Newman, D. A., & Wee, S. (2017, April). Approximation of diversity shrinkage from Pareto weights for diversity-performance tradeoffs. Paper presented at the Annual Conference of the Society for Industrial and Organizational Psychology, Orlando, FL.Google Scholar
Song, Q. C., Newman, D. A., & Wee, S. (2018, April). Enhancing diversity: Pareto-optimal weighting algorithm with regularization. Paper presented at the Annual Conference of the Society for Industrial and Organizational Psychology, Chicago, IL.Google Scholar
Song, Q. C., & Tang, C. (2020, April). Adverse impact reduction for multiple subgroups: A Pareto-optimization approach. In Q. C. Song (Co-Chair) and S. Wee (Co-Chair), Multi-Objective Optimization in the Workplace: Addressing Adverse Impact in Selection. Symposium presentation at the 35th Annual Conference of the Society for Industrial and Organizational Psychology, Austin, TX.Google Scholar
Song, Q. C., Wee, S., & Newman, D. A. (2016, April). Cross-validating Pareto-optimal weights for reducing adverse impact. In P. J. Hanges & J. Y. Park (Co-chairs), New Insights into Adverse Impact: Origination, Motivation, and Scale Weighting. Paper presented at the Annual Conference of the Society for Industrial and Organizational Psychology, Anaheim, CA.Google Scholar
Song, Q. C., Wee, S., & Newman, D. A. (2017). Diversity shrinkage: Cross-validating Pareto-optimal weights to enhance diversity via hiring practices. Journal of Applied Psychology, 102, 1636–1657.CrossRefGoogle ScholarPubMed
Sydell, E., Ferrell, J., Carpenter, J., Frost, C., & Brodbeck, C. C. (2013). Simulation scoring. In Fetzer, M. & Tuzinski, K. (Eds.), Simulations for personnel selection (pp. 83–107). New York, NY: Springer.CrossRefGoogle Scholar
Tam, A. P., Murphy, K. R., & Lyall, J. T. (2004). Can changes in differential dropout rates reduce adverse impact? A computer simulation study of a multi-wave selection system. Personnel Psychology, 57, 905–934.CrossRefGoogle Scholar
Tsang, H. (2010). Improving the adverse impact and validity trade-off in Pareto optimal composites: A comparison of weights developed on contextual vs task performance differences (Doctoral dissertation, University of Central Florida).Google Scholar
Uniform Guidelines on Employee Selection Procedure (1978); 43 FR ___ (August 25, 1978).Google Scholar
Walkowiak, V. S. (2008). An overview of the attorney-client privilege when the client is a corporation. The attorney-client privilege in civil litigation: Protecting and defending confidentiality (pp. 1–56).Google Scholar
Wee, S., Newman, D. A., & Joseph, D. L. (2014). More than g: Selection quality and adverse impact implications of considering second-stratum cognitive abilities. Journal of Applied Psychology, 99, 547–563.10.1037/a0035183CrossRefGoogle Scholar
Wee, S., Newman, D. A., & Song, Q. C. (2015). More than g-factors: Second-stratum factors should not be ignored. Industrial and Organizational Psychology, 8, 482–488.CrossRefGoogle Scholar