Skip to main content Accessibility help
×
Home

Measurement Error Obfuscates Scientific Knowledge: Path to Cumulative Knowledge Requires Corrections for Unreliability and Psychometric Meta-Analyses

Published online by Cambridge University Press:  10 April 2015

Chockalingam Viswesvaran
Affiliation:
Florida International University
Deniz S. Ones
Affiliation:
University of Minnesota-Twin Cities
Frank L. Schmidt
Affiliation:
University of Iowa
Huy Le
Affiliation:
University of Texas-San Antonio
In-Sue Oh
Affiliation:
Temple University
Corresponding
E-mail address:

Abstract

Image of the first page of this article
Type
Commentaries
Copyright
Copyright © Society for Industrial and Organizational Psychology 2014

Access options

Get access to the full version of this content by using one of the access options below.

References

Austin, J. T., & Villanova, P. (1992). The criterion problem: 1917-1992. Journal of Applied Psychology, 77(6), 836874.CrossRefGoogle Scholar
Binning, J. F., & Barrett, G. V. (1989). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74, 478494.CrossRefGoogle Scholar
Borman, W. C., & Brush, D. H. (1993). More progress towards a taxonomy of managerial performance requirements. Human Performance, 6, 121.CrossRefGoogle Scholar
Campbell, J. P., Gasser, M. B., & Oswald, F. L. (1996). The substantive nature of job performance variability. In Murphy, K. R. (Ed.), Individual differences and behavior in organizations (pp. 258299). San Francisco, CA: Jossey-Bass.Google Scholar
Cattin, P. (1980). Estimation of the squared cross-validated multiple correlation of a regression-model. Psychological Bulletin, 87(1), 6365.CrossRefGoogle Scholar
Charles, E. P. (2005). The correction for attenuation due to measurement error: Clarifying concepts and creating confidence sets. Psychological Methods, 10(2), 206226.CrossRefGoogle ScholarPubMed
Connelly, B. S., & Ones, D. S. (2010). An other perspective on personality: Meta-analytic integration of observers' accuracy and predictive validity. Psychological Bulletin, 136, 10921122.CrossRefGoogle ScholarPubMed
Conway, J. M., Jako, R. A., & Goodman, D. F. (1995). A metaanalysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80(5), 565579.CrossRefGoogle Scholar
Dierdorff, E. C., & Wilson, M. A. (2003). A meta-analysis of job analysis reliability. Journal of Applied Psychology, 88(4), 635646.CrossRefGoogle ScholarPubMed
Dilchert, S., Ones, D. S., & Krueger, R. F. (2014). Maladaptive personality constructs, measures, and work behaviors: Scientific background and employment practice recommendations. Industrial and Organizational Psychology: Perspectives on Science and Practice, 7, 98110.CrossRefGoogle Scholar
Facteau, J. D., & Craig, S. B. (2001). Are performance appraisal ratings from different rating sources comparable? Journal of Applied Psychology, 86, 215227.CrossRefGoogle ScholarPubMed
Harter, J. K., Schmidt, F. L., & Hayes, T. L. (2002). Business-unit-level relationships between employee satisfaction, employee engagement, and business outcomes: A meta-analysis. Journal of Applied Psychology, 87, 268279.CrossRefGoogle ScholarPubMed
Hauser, R. M. (2007). Will practitioners benefit from meta analysis? Academy of Management Perspectives, 21(3), 2428.CrossRefGoogle Scholar
Huffcutt, A. I., Culbertson, S., & Weyhrauch, W. S. (2013). Employment interview reliability: New meta-analytic estimates by structure and format. International Journal of Selection and Assessment, 21(3), 264276.CrossRefGoogle Scholar
Hunter, J. E., & Schmidt, F. L. (1982). Quantifying the effects of psychological interventions on employee job performance and workforce productivity. American Psychologist, 38, 473478.CrossRefGoogle Scholar
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.CrossRefGoogle Scholar
Hunter, J. E., Schmidt, F. L., & Le, H. (2006a). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91, 594612.CrossRefGoogle Scholar
Hunter, J. E., Schmidt, F. L., & Le, H. (2006b). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91(3), 594612.CrossRefGoogle Scholar
King, L. M., Hunter, J. E., & Schmidt, F. L. (1980). Halo in a multidimensional forced-choice performance evaluation scale. Journal of Applied Psychology, 65(5), 507516.CrossRefGoogle Scholar
LeBreton, J. M., Scherer, K. T., & James, L. R. (2014). Corrections for criterion reliability in validity generalization: A false prophet in a land of suspended judgment. Industrial and Organizational Psychology: Perspectives on Science and Practice, 7(4), 478500.Google Scholar
Lievens, F. (2001). Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity. Journal of Applied Psychology, 86(2), 255264.CrossRefGoogle ScholarPubMed
Maurer, T. J., Raju, N. S., & Collins, W. C. (1998). Peer and subordinate performance appraisal measurement equivalence. Journal of Applied Psychology, 83(5), 693702. doi: 10.1037/0021-9010.83.5.693CrossRefGoogle Scholar
Mount, M. K., Judge, T. A., Scullen, S. E., et al. (1998). Trait, rater and level effects in 360-degree performance ratings. Personnel Psychology, 51(3), 557576.CrossRefGoogle Scholar
Ones, D. S., & Dilchert, S. (2012). Employee green behaviors. In Jackson, S. E., Ones, D. S., & Dilchert, S. (Eds.), Managing HR for environmental sustainability (pp. 85116). San Francisco, CA: Jossey-Bass/Wiley.Google Scholar
Ones, D. S., & Dilchert, S. (2013). Counterproductive work behaviors: Concepts, measurement, and nomological network. In Geisinger, K. F., Bracken, B. A., Carlson, J. F., Hansen, J.-I. C., Kuncel, N. R., Reise, S. P., & Rodriguez, M. C. (Eds.), APA handbook of testing and assessment in psychology (pp. 643659). Washington, DC: American Psychological Association.Google Scholar
Ones, D. S., Dilchert, S., & Viswesvaran, C. (2012). Cognitive abilities. In Schmitt, N. (Ed.), Oxford handbook of personnel assessment and selection (pp. 179224). New York, NY: Oxford University Press.Google Scholar
Ones, D. S., Viswesvaran, C., & Dilchert, S. (2005). Cognitive ability in personnel selection decisions. In Evers, A., Voskuijl, O., & Anderson, N. (Eds.), Handbook of selection (pp. 143173). Oxford, England: Blackwell.Google Scholar
Ones, D. S., Viswesvaran, C., & Schmidt, F. L. (2008). No new terrain: Reliability and construct validity of job performance ratings. Industrial and Organizational Psychology, 1, 174179.CrossRefGoogle Scholar
Osterman, P. (2007). Comment on Le, Oh, Shaffer, and Schmidt. Academy of Management Perspectives, 21(3), 1618.CrossRefGoogle Scholar
Ostroff, C., & Harrison, D. A. (1999). Meta-analysis, level of analysis, and best estimates of population correlations: Cautions for interpreting meta-analytic results in organizational behavior. Journal of Applied Psychology, 84(2), 260270.CrossRefGoogle Scholar
Putka, D. J., & Hoffman, B. J. (in press). “The” reliability of performance ratings equals 0.52. In Lance, C. E., & Vandenberg, R. J. (Eds.), More statistical and methodological myths and urban legends. New York, NY: Taylor & Francis.Google Scholar
Putka, D. J., Le, H., McCloy, R. A., & Diaz, T. (2008). Ill-structured measurement designs in organizational research: Implications for estimating interrater reliability. Journal of Applied Psychology, 93, 959981.CrossRefGoogle ScholarPubMed
Rothstein, H. R. (1990). Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology, 75, 322327.CrossRefGoogle Scholar
Sackett, P. R., Laczo, R. M., & Arvey, R. D. (2002). The effects of range restriction on estimates of criterion interrater reliability: Implications for validation research. Personnel Psychology, 55(4), 807825.CrossRefGoogle Scholar
Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., De Fruyt, F., & Rolland, J. P. (2003). A meta-analytic study of general mental ability validity for different occupations in the European Community. Journal of Applied Psychology, 88, 10681081.CrossRefGoogle ScholarPubMed
Salgado, J. F., & Moscoso, S. (1996). Meta-analysis of interrater reliability of job performance ratings. Perceptual and Motor Skills, 83, 11951201.CrossRefGoogle Scholar
Salgado, J. F., Moscoso, S., & Lado, M. (2003). Test-retest reliability of job performance dimensions in managers. International Journal of Selection and Assessment, 11, 98101.CrossRefGoogle Scholar
Salgado, J. F., & Tauriz, G. (2014). The five-factor model, forced-choice personality inventories and performance: A comprehensive meta-analysis of academic and occupational validity studies. European Journal of Work and Organizational Psychology, 23, 330.CrossRefGoogle Scholar
Schmidt, F. L., & Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62(5), 529540.CrossRefGoogle Scholar
Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1(2), 199223.CrossRefGoogle Scholar
Schmidt, F. L., & Hunter, J. E. (2014). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: Sage.Google Scholar
Schmidt, F. L., Hunter, J. E., & Outerbridge, A. N. (1986). Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of performance. Journal of Applied Psychology, 71, 432439.CrossRefGoogle Scholar
Schmidt, F. L., & Kaplan, L. B. (1971). Composite versus multiple criteria: A review and resolution of the controversy. Personnel Psychology, 24, 419434.CrossRefGoogle Scholar
Schmidt, F. L., Le, H., Oh, I., & Schafer, J. (2007). General mental ability, job performance and red herrings: Responses to Osterman, Huaser and Schmitt. Academy of Management Perspectives, 21, 6476.CrossRefGoogle Scholar
Schmidt, F. L., Viswesvaran, C., & Ones, D. S. (2000). Reliability is not validity and validity is not reliability. Personnel Psychology, 53, 901912.CrossRefGoogle Scholar
Schmitt, N. (2007). The value of personnel selection: Reflections on some remarkable claims. Academy of Management Perspectives, 21(3), 1923.CrossRefGoogle Scholar
Shen, W., Beatty, A. S., & Sackett, P. R. (2009, September). Further moderators of the interrater reliability of supervisory ratings of job performance. Washington, DC: Personnel Testing Council of Metropolitan Washington Newsletter.CrossRefGoogle Scholar
Society for Industrial and Organizational Psychology (SIOP). (2003). Principles for the validation and use of personnel selection procedures (4th ed.). Bowling Green, OH: Society for Industrial and Organizational Psychology.Google Scholar
Thompson, B. (1992). Two and one-half decades of leadership in measurement and evaluation. Journal of Counseling and Development, 70, 434438.CrossRefGoogle Scholar
Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 620.CrossRefGoogle Scholar
Viswesvaran, C., & Ones, D. S. (2000a). Perspectives on models of job performance. International Journal of Selection and Assessment, 8, 216227.CrossRefGoogle Scholar
Viswesvaran, C., & Ones, D. S. (2000b). Measurement error in “Big Five Factors” of personality assessment: Reliability generalization across studies and measures. Educational and Psychological Measurement, 60, 224235.CrossRefGoogle Scholar
Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557574.CrossRefGoogle Scholar
Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2002). The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty. Journal of Applied Psychology, 87, 345354.CrossRefGoogle ScholarPubMed
Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108131.CrossRefGoogle Scholar
Voskuijl, O. F., & vanSliedregt, T. (2002). Determinants of interrater reliability of job analysis: Ameta-analysis. European Journal of Psychological Assessment, 18(1), 5262.CrossRefGoogle Scholar
Whitman, D. S., Van Rooy, D. L., & Viswesvaran, C. (2010). Satisfaction, citizenship behaviors and performance in work units: A meta-analysis of collective construct relations. Personnel Psychology, 63, 4181.CrossRefGoogle Scholar

Altmetric attention score

Full text views

Full text views reflects PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views.

Total number of HTML views: 0
Total number of PDF views: 126 *
View data table for this chart

* Views captured on Cambridge Core between September 2016 - 16th January 2021. This data will be updated every 24 hours.

Hostname: page-component-77fc7d77f9-xz9qf Total loading time: 1.039 Render date: 2021-01-16T22:03:23.489Z Query parameters: { "hasAccess": "0", "openAccess": "0", "isLogged": "0", "lang": "en" } Feature Flags last update: Sat Jan 16 2021 21:53:08 GMT+0000 (Coordinated Universal Time) Feature Flags: { "metrics": true, "metricsAbstractViews": false, "peerReview": true, "crossMark": true, "comments": true, "relatedCommentaries": true, "subject": true, "clr": true, "languageSwitch": true, "figures": false, "newCiteModal": false, "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true }

Send article to Kindle

To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Measurement Error Obfuscates Scientific Knowledge: Path to Cumulative Knowledge Requires Corrections for Unreliability and Psychometric Meta-Analyses
Available formats
×

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Measurement Error Obfuscates Scientific Knowledge: Path to Cumulative Knowledge Requires Corrections for Unreliability and Psychometric Meta-Analyses
Available formats
×

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Measurement Error Obfuscates Scientific Knowledge: Path to Cumulative Knowledge Requires Corrections for Unreliability and Psychometric Meta-Analyses
Available formats
×
×

Reply to: Submit a response


Your details


Conflicting interests

Do you have any conflicting interests? *