Hostname: page-component-8448b6f56d-dnltx Total loading time: 0 Render date: 2024-04-23T19:29:40.777Z Has data issue: false hasContentIssue false
  • English
  • Français

Measurement Error Obfuscates Scientific Knowledge: Path to Cumulative Knowledge Requires Corrections for Unreliability and Psychometric Meta-Analyses

Published online by Cambridge University Press:  10 April 2015

Chockalingam Viswesvaran ,
Deniz S. Ones ,
Frank L. Schmidt ,
Huy Le  and
In-Sue Oh
Chockalingam Viswesvaran*
Affiliation:
Florida International University
Deniz S. Ones
Affiliation:
University of Minnesota-Twin Cities
Frank L. Schmidt
Affiliation:
University of Iowa
Huy Le
Affiliation:
University of Texas-San Antonio
In-Sue Oh
Affiliation:
Temple University
*
E-mail: vish@fiu.edu, Address: Department of Psychology, Florida International University, University Park, Miami FL 33199
Get access Rights & Permissions [Opens in a new window]

Abstract

An abstract is not available for this content so a preview has been provided. Please use the Get access link above for information on how to access this content.
Image of the first page of this content. For PDF version, please use the ‘Save PDF’ preceeding this image.'
Type
Commentaries
Information
Industrial and Organizational Psychology , Volume 7 , Issue 4 , December 2014 , pp. 507 - 518
Copyright
Copyright © Society for Industrial and Organizational Psychology 2014

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Austin, J. T., & Villanova, P. (1992). The criterion problem: 1917-1992. Journal of Applied Psychology, 77(6), 836–874.Google Scholar
Binning, J. F., & Barrett, G. V. (1989). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74, 478–494.CrossRefGoogle Scholar
Borman, W. C., & Brush, D. H. (1993). More progress towards a taxonomy of managerial performance requirements. Human Performance, 6, 1–21.Google Scholar
Campbell, J. P., Gasser, M. B., & Oswald, F. L. (1996). The substantive nature of job performance variability. In Murphy, K. R. (Ed.), Individual differences and behavior in organizations (pp. 258–299). San Francisco, CA: Jossey-Bass.Google Scholar
Cattin, P. (1980). Estimation of the squared cross-validated multiple correlation of a regression-model. Psychological Bulletin, 87(1), 63–65.CrossRefGoogle Scholar
Charles, E. P. (2005). The correction for attenuation due to measurement error: Clarifying concepts and creating confidence sets. Psychological Methods, 10(2), 206–226.Google Scholar
Connelly, B. S., & Ones, D. S. (2010). An other perspective on personality: Meta-analytic integration of observers' accuracy and predictive validity. Psychological Bulletin, 136, 1092–1122.Google Scholar
Conway, J. M., Jako, R. A., & Goodman, D. F. (1995). A metaanalysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80(5), 565–579.Google Scholar
Dierdorff, E. C., & Wilson, M. A. (2003). A meta-analysis of job analysis reliability. Journal of Applied Psychology, 88(4), 635–646.Google Scholar
Dilchert, S., Ones, D. S., & Krueger, R. F. (2014). Maladaptive personality constructs, measures, and work behaviors: Scientific background and employment practice recommendations. Industrial and Organizational Psychology: Perspectives on Science and Practice, 7, 98–110.Google Scholar
Facteau, J. D., & Craig, S. B. (2001). Are performance appraisal ratings from different rating sources comparable? Journal of Applied Psychology, 86, 215–227.CrossRefGoogle ScholarPubMed
Harter, J. K., Schmidt, F. L., & Hayes, T. L. (2002). Business-unit-level relationships between employee satisfaction, employee engagement, and business outcomes: A meta-analysis. Journal of Applied Psychology, 87, 268–279.Google Scholar
Hauser, R. M. (2007). Will practitioners benefit from meta analysis? Academy of Management Perspectives, 21(3), 24–28.Google Scholar
Huffcutt, A. I., Culbertson, S., & Weyhrauch, W. S. (2013). Employment interview reliability: New meta-analytic estimates by structure and format. International Journal of Selection and Assessment, 21(3), 264–276.Google Scholar
Hunter, J. E., & Schmidt, F. L. (1982). Quantifying the effects of psychological interventions on employee job performance and workforce productivity. American Psychologist, 38, 473–478.Google Scholar
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.Google Scholar
Hunter, J. E., Schmidt, F. L., & Le, H. (2006a). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91, 594–612.Google Scholar
Hunter, J. E., Schmidt, F. L., & Le, H. (2006b). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91(3), 594–612.Google Scholar
King, L. M., Hunter, J. E., & Schmidt, F. L. (1980). Halo in a multidimensional forced-choice performance evaluation scale. Journal of Applied Psychology, 65(5), 507–516.Google Scholar
LeBreton, J. M., Scherer, K. T., & James, L. R. (2014). Corrections for criterion reliability in validity generalization: A false prophet in a land of suspended judgment. Industrial and Organizational Psychology: Perspectives on Science and Practice, 7(4), 478–500.Google Scholar
Lievens, F. (2001). Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity. Journal of Applied Psychology, 86(2), 255–264.Google Scholar
Maurer, T. J., Raju, N. S., & Collins, W. C. (1998). Peer and subordinate performance appraisal measurement equivalence. Journal of Applied Psychology, 83(5), 693–702. doi: 10.1037/0021-9010.83.5.693Google Scholar
Mount, M. K., Judge, T. A., Scullen, S. E., et al. (1998). Trait, rater and level effects in 360-degree performance ratings. Personnel Psychology, 51(3), 557–576.CrossRefGoogle Scholar
Ones, D. S., & Dilchert, S. (2012). Employee green behaviors. In Jackson, S. E., Ones, D. S., & Dilchert, S. (Eds.), Managing HR for environmental sustainability (pp. 85–116). San Francisco, CA: Jossey-Bass/Wiley.Google Scholar
Ones, D. S., & Dilchert, S. (2013). Counterproductive work behaviors: Concepts, measurement, and nomological network. In Geisinger, K. F., Bracken, B. A., Carlson, J. F., Hansen, J.-I. C., Kuncel, N. R., Reise, S. P., & Rodriguez, M. C. (Eds.), APA handbook of testing and assessment in psychology (pp. 643–659). Washington, DC: American Psychological Association.Google Scholar
Ones, D. S., Dilchert, S., & Viswesvaran, C. (2012). Cognitive abilities. In Schmitt, N. (Ed.), Oxford handbook of personnel assessment and selection (pp. 179–224). New York, NY: Oxford University Press.Google Scholar
Ones, D. S., Viswesvaran, C., & Dilchert, S. (2005). Cognitive ability in personnel selection decisions. In Evers, A., Voskuijl, O., & Anderson, N. (Eds.), Handbook of selection (pp. 143–173). Oxford, England: Blackwell.Google Scholar
Ones, D. S., Viswesvaran, C., & Schmidt, F. L. (2008). No new terrain: Reliability and construct validity of job performance ratings. Industrial and Organizational Psychology, 1, 174–179.Google Scholar
Osterman, P. (2007). Comment on Le, Oh, Shaffer, and Schmidt. Academy of Management Perspectives, 21(3), 16–18.Google Scholar
Ostroff, C., & Harrison, D. A. (1999). Meta-analysis, level of analysis, and best estimates of population correlations: Cautions for interpreting meta-analytic results in organizational behavior. Journal of Applied Psychology, 84(2), 260–270.Google Scholar
Putka, D. J., & Hoffman, B. J. (in press). “The” reliability of performance ratings equals 0.52. In Lance, C. E., & Vandenberg, R. J. (Eds.), More statistical and methodological myths and urban legends. New York, NY: Taylor & Francis.Google Scholar
Putka, D. J., Le, H., McCloy, R. A., & Diaz, T. (2008). Ill-structured measurement designs in organizational research: Implications for estimating interrater reliability. Journal of Applied Psychology, 93, 959–981.Google Scholar
Rothstein, H. R. (1990). Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology, 75, 322–327.Google Scholar
Sackett, P. R., Laczo, R. M., & Arvey, R. D. (2002). The effects of range restriction on estimates of criterion interrater reliability: Implications for validation research. Personnel Psychology, 55(4), 807–825.Google Scholar
Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., De Fruyt, F., & Rolland, J. P. (2003). A meta-analytic study of general mental ability validity for different occupations in the European Community. Journal of Applied Psychology, 88, 1068–1081.Google Scholar
Salgado, J. F., & Moscoso, S. (1996). Meta-analysis of interrater reliability of job performance ratings. Perceptual and Motor Skills, 83, 1195–1201.Google Scholar
Salgado, J. F., Moscoso, S., & Lado, M. (2003). Test-retest reliability of job performance dimensions in managers. International Journal of Selection and Assessment, 11, 98–101.CrossRefGoogle Scholar
Salgado, J. F., & Tauriz, G. (2014). The five-factor model, forced-choice personality inventories and performance: A comprehensive meta-analysis of academic and occupational validity studies. European Journal of Work and Organizational Psychology, 23, 3–30.Google Scholar
Schmidt, F. L., & Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62(5), 529–540.Google Scholar
Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1(2), 199–223.CrossRefGoogle Scholar
Schmidt, F. L., & Hunter, J. E. (2014). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: Sage.Google Scholar
Schmidt, F. L., Hunter, J. E., & Outerbridge, A. N. (1986). Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of performance. Journal of Applied Psychology, 71, 432–439.Google Scholar
Schmidt, F. L., & Kaplan, L. B. (1971). Composite versus multiple criteria: A review and resolution of the controversy. Personnel Psychology, 24, 419–434.Google Scholar
Schmidt, F. L., Le, H., Oh, I., & Schafer, J. (2007). General mental ability, job performance and red herrings: Responses to Osterman, Huaser and Schmitt. Academy of Management Perspectives, 21, 64–76.Google Scholar
Schmidt, F. L., Viswesvaran, C., & Ones, D. S. (2000). Reliability is not validity and validity is not reliability. Personnel Psychology, 53, 901–912.Google Scholar
Schmitt, N. (2007). The value of personnel selection: Reflections on some remarkable claims. Academy of Management Perspectives, 21(3), 19–23.Google Scholar
Shen, W., Beatty, A. S., & Sackett, P. R. (2009, September). Further moderators of the interrater reliability of supervisory ratings of job performance. Washington, DC: Personnel Testing Council of Metropolitan Washington Newsletter.Google Scholar
Society for Industrial and Organizational Psychology (SIOP). (2003). Principles for the validation and use of personnel selection procedures (4th ed.). Bowling Green, OH: Society for Industrial and Organizational Psychology.Google Scholar
Thompson, B. (1992). Two and one-half decades of leadership in measurement and evaluation. Journal of Counseling and Development, 70, 434–438.Google Scholar
Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6–20.CrossRefGoogle Scholar
Viswesvaran, C., & Ones, D. S. (2000a). Perspectives on models of job performance. International Journal of Selection and Assessment, 8, 216–227.Google Scholar
Viswesvaran, C., & Ones, D. S. (2000b). Measurement error in “Big Five Factors” of personality assessment: Reliability generalization across studies and measures. Educational and Psychological Measurement, 60, 224–235.Google Scholar
Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557–574.Google Scholar
Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2002). The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty. Journal of Applied Psychology, 87, 345–354.Google Scholar
Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108–131.CrossRefGoogle Scholar
Voskuijl, O. F., & vanSliedregt, T. (2002). Determinants of interrater reliability of job analysis: Ameta-analysis. European Journal of Psychological Assessment, 18(1), 52–62.Google Scholar
Whitman, D. S., Van Rooy, D. L., & Viswesvaran, C. (2010). Satisfaction, citizenship behaviors and performance in work units: A meta-analysis of collective construct relations. Personnel Psychology, 63, 41–81.CrossRefGoogle Scholar