Hostname: page-component-8448b6f56d-42gr6 Total loading time: 0 Render date: 2024-04-24T13:03:14.096Z Has data issue: false hasContentIssue false
  • English
  • Français

At Sea With Synthetic Validity

Published online by Cambridge University Press:  07 January 2015

Piers Steel ,
Jeff W. Johnson ,
P. Richard Jeanneret ,
Charles A. Scherbaum ,
Calvin C. Hoffman  and
Jeff Foster
Piers Steel*
Affiliation:
University of Calgary
Jeff W. Johnson
Affiliation:
Personnel Decisions Research Institutes
P. Richard Jeanneret
Affiliation:
Valtera Corporation
Charles A. Scherbaum
Affiliation:
Baruch College
Calvin C. Hoffman
Affiliation:
Los Angeles County Sheriff's Department and Alliant University
Jeff Foster
Affiliation:
Hogan Assessment Systems
*
E-mail: piers.steel@haskayne.ucalgary.ca, Address: Piers Steel, Haskayne School of Business, University of Calgary, SH444 - 500 University Drive, N.W. Calgary, Alberta, Canada T2N 1N4; Jeff W. Johnson, Personnel Decisions Research Institutes; Charles A. Scherbaum, Department of Psychology, Baruch College; Calvin C. Hoffman, Los Angeles County Sheriff's Department and Alliant University; P. Richard Jeanneret, Valtera Corporation; Jeff Foster, Hogan Assessment Systems.
Get access Rights & Permissions [Opens in a new window]

Abstract

We expected that the commentary process would provide valuable feedback to improve our ideas and identify potential obstacles, and we were not disappointed. The commentaries were generally in agreement that synthetic validity is a good idea, although we also received a fair amount of suggestions for improvements, conditional or tempered praise, and explicitly critical comments. We address the concerns that were raised and conclude that we should move forward with developing a large-scale synthetic validity database, incorporating the suggestions of some of the commentators.

Type
Response
Information
Industrial and Organizational Psychology , Volume 3 , Issue 3 , September 2010 , pp. 371 - 383
Copyright
Copyright © Society for Industrial and Organizational Psychology 2010 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Alf, E. F. Jr., & Graf, R. G. (2002). A new maximum likelihood estimator of the population squared multiple correlation. Journal of Educational and Behavioral Statistics, 27, 223–235.Google Scholar
Bartram, D., Warr, P., & Brown, A. (2010). Let's focus on two-stage alignment not just on overall performance. Industrial and Organizational Psychology: Perspectives on Science and Practice, 3, 335–339.Google Scholar
Bertua, C., Anderson, N., & Salgado, J. F. (2005). The predictive validity of cognitive ability tests: A UK meta-analysis. Journal of Occupational and Organizational Psychology, 78, 387–409.Google Scholar
Converse, P. D., Oswald, F. L., Gillespie, M. A., Field, K. A., & Bizot, E. B. (2004). Matching individuals to occupations using abilities and the O*NET. Personnel Psychology, 57, 451–487.Google Scholar
Dudley, N., Orvis, K., Lebiecki, J., & Cortina, J. (2006). A meta-analytic investigation of conscientiousness in the prediction of job performance: Examining the intercorrelations and the incremental validity of narrow traits. Journal of Applied Psychology, 91, 40–57.Google Scholar
Gibson, W., & Caplinger, J. (2007). Transportation of validation results. In McPhail, S. M. (Ed.), Alternative validation strategies: Developing new and leveraging existing validity evidence (pp. 29–81). San Francisco: John Wiley & Sons.Google Scholar
Griffeth, R. W., Hom, P. W., & Gaertner, S. (2000). A meta-analysis of antecedents and correlates of employee turnover: Update, moderator tests, and research implications for the next millennium. Journal of Management, 26, 463–488.Google Scholar
Harvey, R. J. (2010). Motor oil or snake oil: synthetic validity is a tool not a panacea. Industrial and Organizational Psychology: Perspectives on Science and Practice, 3, 351–355.Google Scholar
Hoffman, B., Lance, C. E., Bynum, D., & Gentry, W. (2010). Rater source effects are alive and well after all. Personnel Psychology, 63, 119–151.Google Scholar
Hollweg, L. (2010). Synthetic oil is better for whom? Industrial and Organizational Psychology: Perspectives on Science and Practice, 3, 363–365.Google Scholar
Hunter, J. E. (1983). Validity generalization for 12,000 jobs: An application of synthetic validity and validity generalization to the General Aptitude Test Battery (GATB). Washington, DC: U.S. Department of Labor Employment Service.Google Scholar
Hunter, J. E., & Schmidt, F. L. (1982). Fitting people to jobs: The impact of personnel selection on national productivity. In Dunnette, M. D., & Fleishman, E. A. (Eds.), Human performance and productivity: Vol. 1. Human capability assessment (pp. 223–272). Beverly Hills, CA: Sage.Google Scholar
Hülsheger, U. R., Maier, G. W., & Stumpp, T. (2007). Validity of general mental ability for the prediction of job performance and training success in Germany: A meta-analysis. International Journal of Selection and Assessment, 15, 3–18.Google Scholar
Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72–98.Google Scholar
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting for error and bias in research findings across studies (2nd ed.). Thousand Oaks, CA: Sage.Google Scholar
Jeanneret, P. R. (1992). Applications of job component/synthetic validity to construct validity. Human Performance, 5, 81–96.Google Scholar
Johnson, J. W., Steel, P., Scherbaum, C. A., Hoffman, C. C., Jeanneret, P. R., & Foster, J. (2010). Validation is like motor oil: Synthetic is better. Industrial and Organizational Psychology: Perspectives on Science and Practice, 3, 305–328.Google Scholar
Judge, T. A., Heller, D., & Mount, M. K. (2002). Five-factor model of personality and job satisfaction: A meta-analysis. Journal of Applied Psychology, 87, 530–541. doi: 10.1037//0021-9010.87.3.530.Google Scholar
King, L. M., Hunter, J. E., & Schmidt, F. L. (1980). Halo in a multidimensional forced choice performance and evaluation scale. Journal of Applied Psychology, 65, 502–516.Google Scholar
Koopman, R. (1988). On the sensitivity of a composite to its weights. Psychometrika, 53, 547–552.Google Scholar
Kristof-Brown, A. L., Zimmerman, R. D., & Johnson, E. C. (2006). Perceived applicant fit: Distinguishing between recruiters' perceptions of person-job and person-organization fit. Personnel Psychology, 53, 643–671.Google Scholar
Lance, C. E., Baranik, L. E., Lau, A. R., & Scharlau, E. A. (2009). If it ain't trait it must be method: (Mis)application of the multitrait-multimethod methodology in organizational research. In Lance, C. E., & Vandenberg, R. J. (Eds.), Statistical and methodological myths and urban legends: Received doctrine, verity, and fable in organizational and social research (pp. 339–362). New York: Routledge.Google Scholar
Lance, C. E., Dawson, B., Birkelbach, D., & Hoffman, B. J. (2010). Method effects, measurement error, and substantive conclusions. Organizational Research Methods. doi: 10.1177/1094428109352 528.Google Scholar
Lawshe, C. H. (1952). Employee selection. Personnel Psychology, 6, 31–34.Google Scholar
Lubinski, D., & Humphreys, L. (1996). Seeing the forest from the trees: When predicting the behavior or status of groups, correlate means. Psychology, Public Policy, and Law, 2, 363–376.Google Scholar
McCloy, R. A., Putka, D. J., & Gibby, R. E. (2010). Developing an online synthetic validation tool. Industrial and Organizational Psychology: Perspectives on Science and Practice, 3, 366–370.Google Scholar
Murphy, K. R. (2000). Impact of assessments of validity generalization and situational specificity on the science and practice of personnel selection. International Journal of Selection and Assessment, 8, 194–206.Google Scholar
Murphy, K. R. (2009a). Content validation is useful for many things, but validity isn't one of them. Perspectives on Science and Practice, 2, 453–464.Google Scholar
Murphy, K. R. (2009b). Is content-related validation useful for in validating selection tests? Perspectives on Science and Practice, 2, 517–526.Google Scholar
Murphy, K. R. (2010). Synthetic validity: A great idea whose time never came. Industrial and Organizational Psychology: Perspectives on Science and Practice, 3, 356–359.Google Scholar
Oswald, F. L., & Hough, L. M. (2010). How synthetic validation contributes to personnel selection. Industrial and Organizational Psychology: Perspectives on Science and Practice, 3, 329–334.Google Scholar
Peck, D. (2010, March). How a jobless era will transform America. Atlantic Monthly, 305, 42–56.Google Scholar
Primoff, E., & Fine, S. (1988). A history of job analysis. In Gael, S. (Ed.), The job analysis handbook for business, industry and government (pp. 14–29). Toronto: John Wiley & Sons.Google Scholar
Pritchard, R. D., Paquin, A. R., DeCuir, A. D., McCormick, M. J., & Bly, P. R. (2002). Measuring and improving organizational productivity: An overview of ProMes, the Productivity Measurement and Enhancement system. In Pritchard, R. D., Holling, H., Lammers, F., & Clark, B. D. (Eds.), Improving organizational performance with the Productivity Measurement and Enhancement system: An international collaboration (pp. 3–50). Huntington, NY: Nova Science.Google Scholar
Ree, M., Carretta, T., & Earles, J. (1998). In top-down decisions, weighting variables does not matter: A consequence of Wilks' theorem. Organizational Research Methods, 1, 407–420.Google Scholar
Rothstein, M. G., & Goffin, R. D. (2006). The use of personality measures in personnel selection: What does current research support? Human Resource Management Review, 16, 155–180.Google Scholar
Russell, C. J. (2010). Better at what? Industrial and Organizational Psychology: Perspectives on Science and Practice, 3, 340–343.Google Scholar
Schmidt, F. L., Law, K., Hunter, J. E., Rothstein, J. R., Pearlman, K., & McDaniel, M. A. (1993). Refinements in validity generalization procedures: Implications for the situational specificity hypothesis. Journal of Applied Psychology, 78, 3–13.Google Scholar
Schmidt, F. L., & Oh, I.-S. (2010). Can synthetic validity methods achieve discriminant validity? Industrial and Organizational Psychology: Perspectives on Science and Practice, 3, 344–350.Google Scholar
Schneider, R. J., Hough, L. M., & Dunnette, M. D. (1996). Broadsided by broad traits: How to sink science in five dimensions or less. Journal of Organizational Behavior, 17, 639–655.Google Scholar
Society for Industrial and Organizational Psychology (2003). Principles for the validation and use of personnel selection procedures. Bowling Green, OH: SIOP.Google Scholar
Steel, P., & Kammeyer-Mueller, J. (2002). Comparing meta-analytic moderator search techniques under realistic conditions. Journal of Applied Psychology, 87, 96–111.Google Scholar
Steel, P., & Kammeyer-Mueller, J. (2008). Bayesian variance estimation for meta-analysis: Quantifying our uncertainty. Organizational Research Methods, 11, 54–78.Google Scholar
Steel, P., & Kammeyer-Mueller, J. (2009). Using a meta-analytic perspective to enhance job component validation. Personnel Psychology, 62, 533–552.Google Scholar
Steel, P., Huffcutt, A., & Kammeyer-Muller, J. (2006). From the work one knows the worker: A systematic review of the challenges, solutions, and steps to creating synthetic validity. International Journal of Selection and Assessment, 14, 16–36.Google Scholar
Tett, R. P., & Christiansen, N. D. (2007). Personality tests at the crossroads: A response to Morgeson, Campion, Dipboye, Hollenbeck, Murphy, and Schmitt (2007). Personnel Psychology, 60, 967–993.Google Scholar
Vancouver, J. B. (2010). Improving I-O science through synthetic validity. Industrial and Organizational Psychology: Perspectives on Science and Practice, 3, 360–362.Google Scholar
Wheelan, C. (2002). Naked economics: Undressing the dismal science. New York. W. W. Norton.Google Scholar