Hostname: page-component-77c89778f8-vsgnj Total loading time: 0 Render date: 2024-07-18T22:23:16.996Z Has data issue: false hasContentIssue false
  • English
  • Français

Where Have We Been, How Did We Get There, and Where Shall We Go?

Published online by Cambridge University Press:  07 January 2015

Charles E. Lance
Charles E. Lance*
Affiliation:
The University of Georgia
*
E-mail: clance@uga.edu, Address: Department of Psychology, The University of Georgia, Athens, GA 30602-3013
Get access Rights & Permissions [Opens in a new window]

Abstract

Commentators expressed a wide variety of views on my evaluation of the state of assessment center (AC) research and practice. In this response, I first trace the evolution of the construct validity paradox “urban legend.” Next, I consider the commentators’ comments as they relate to (a) my recommendation to abandon dimension-based ACs in lieu of task- or role-based structures (b) my recommendation to discontinue design fix attempts toward making ACs conform to multitrait–multimethod construct validity criteria, and (c) considerations of construct validity and validation evidence. Finally, I offer some directions for future AC research and practice.

Type
Response
Information
Industrial and Organizational Psychology , Volume 1 , Issue 1 , March 2008 , pp. 140 - 146
Copyright
Copyright © Society for Industrial and Organizational Psychology 2008 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

*

Department of Psychology, The University of Georgia

This work was supported in part by National Institute on Aging Grant 5R01AG15321-10, Gail M. Williamson principal investigator and National Institute on Drug Abuse Grant R01DA019460-01A1, Lillian Eby, principal investigator

References

Arthur, W. Jr., Day, E. A., & Woehr, D. J. (2008). Mend it, don’t end it: An alternate view of assessment center construct-related validity evidence. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 105–111.10.1111/j.1754-9434.2007.00019.xGoogle Scholar
Arthur, W. Jr., Woehr, D. J., & Maldegan, R. (2000). Convergent and discriminant validity of assessment center dimensions: A conceptual and empirical re-examination of the assessment center construct-related validity paradox. Journal of Management, 26, 813–835.Google Scholar
Bergman, A., & Bergman, M. (1973). The way we were [Recorded by B. Streisand]. On The way we were [CD]. Tokyo: Sony. (2002).Google Scholar
Binning, J. F., & Barrett, G. V. (1988). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74, 478–494.10.1037/0021-9010.74.3.478Google Scholar
Borsboom, D., Mellenbergh, G. J., & Van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061–1071.10.1037/0033-295X.111.4.1061Google Scholar
Bowler, M. C., & Woehr, D. J. (2006). A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings. Journal of Applied Psychology, 91, 1114–1124.10.1037/0021-9010.91.5.1114Google Scholar
Brannick, M. T. (2008). Back to basics of test construction and scoring. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 131–133.10.1111/j.1754-9434.2007.00025.xGoogle Scholar
Bray, D. W., Campbell, R. J., & Grant, D. L. (1974). Formative years in business: A long-term AT&T study of managerial lives. New York: Wiley.Google Scholar
Bray, D. W., & Grant, D. L. (1966). The assessment center in the measurement of potential for business management. Psychological Monographs, 80 (17, Whole No. 625), 1–27.10.1037/h0093895Google Scholar
Campbell, J. P. (1990). Modeling the performance prediction problem in industrial and organizational psychology. In Dunnette, M. D. & Hough, L. M. (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 1, pp. 687–732). Palo Alto, CA: Consulting Psychologists Press.Google Scholar
Campbell, J. P., McHenry, J. J., & Wise, L. L. (1990). Modeling job performance in a population of jobs. Personnel Psychology, 43, 313–333.10.1111/j.1744-6570.1990.tb01561.xGoogle Scholar
Cervone, D., & Shoda, Y. (1999). Social-cognitive theories and the coherence of personality. In Cervone, D. & Shoda, Y. (Eds.), The coherence of personality: Social-cognitive bases of personality consistency, variability, and organization (pp. 3–33). New York: Guilford.Google Scholar
Connelly, B. S., Ones, D. S., Ramesh, A., & Goff, M. (2008). A pragmatic view of assessment center exercises and dimensions. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 121–124.10.1111/j.1754-9434.2007.00022.xGoogle Scholar
Cowan, N. (2000). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87–114.10.1017/S0140525X01003922Google Scholar
Dunnette, M. D. (1971). The assessment of managerial talent. In McReynolds, P. (Ed.), Advances in psychological assessment (Vol. 2, pp. 79–108). Palo Alto, CA: Science and Behavior Books.Google Scholar
Ekehammer, B. (1974). Interactionism in personality from a historical perspective. Psychological Bulletin, 81, 1026–1048.10.1037/h0037457Google Scholar
Howard, A. (1997). A reassessment of assessment centers: Challenges for the 21st century. Journal of Social Behavior and Personality, 12(5), 13–52.Google Scholar
Howard, A. (2008). Making assessment centers work the way they’re supposed to. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 98–104.10.1111/j.1754-9434.2007.00018.xGoogle Scholar
Jackson, D. J. R., Barney, A. R., Stillman, J. A., & Kirkley, B. (2007). When traits are behaviors: The relationship between behavioral responses and trait-based overall assessment center ratings. Human Performance, 20, 415–432.Google Scholar
Jackson, D. J. R., Stillman, J. A., & Atkins, S. G. (2005). Rating tasks versus dimensions in assessment centers: A psychometric comparison. Human Performance, 18, 213–241.10.1207/s15327043hup1803_2Google Scholar
Jones, R. G., & Klimoski, R. J. (2008). Narrow standards for efficacy and the research playground: Why either–or conclusions do not help. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 137–139.10.1111/j.1754-9434.2007.00027.xGoogle Scholar
Joyce, L. W., Thayer, P. W., & Pond, S. B. III (1994). Managerial functions: An alternative to traditional assessment center dimensions? Personnel Psychology, 47, 109–121.10.1111/j.1744-6570.1994.tb02412.xGoogle Scholar
Kelly, T. L. (1927). Interpretations of educational measurements. New York: Macmillan.Google Scholar
Lance, C. E., Baranik, L. E., Lau, A. R., & Scharlau, E. A. (in press). If it ain’t trait it must be method: (Mis) application of the multitrait-multimethod methodology in organizational research. In Lance, C. E. & Vandenberg, R. J. (Eds.), Statistical and methodological myths and urban legends: Received doctrine, verity, and fable in organizational and social research. New York: Routledge.10.4324/9780203867266Google Scholar
Lance, C. E., Foster, M. R., Drollinger, S. M., Sorensen, K. L., Gentry, W. A., & Nemeth, Y. M. (2007, April). A comparison of task-based versus dimension-based scoring procedures in an operational assessment center (AC). Paper presented at the meeting of the Society for Industrial and Organizational Psychology, New York, NY.Google Scholar
Lance, C. E., Foster, M. R., Gentry, W. A., & Thoresen, J. D. (2004). Assessor cognitive processes in an operational assessment center. Journal of Applied Psychology, 89, 22–35.10.1037/0021-9010.89.1.22Google Scholar
Lance, C. E., Foster, M. R., Nemeth, Y. M., Gentry, W. A., & Drollinger, S. (2007). Extending the nomological network of assessment center construct validity: Prediction of cross-situationally consistent and specific aspects of assessment center performance. Human Performance, 20, 345–362.Google Scholar
Lance, C. E., Lambert, T. A., Gewin, A. G., Lievens, F., & Conway, J. M. (2004). Revised estimates of dimension and exercise variance components in assessment center post-exercise dimension ratings. Journal of Applied Psychology, 89, 377–385.10.1037/0021-9010.89.2.377Google Scholar
Lance, C. E., Newbolt, W. H., Gatewood, R. D., Foster, M. R., French, N., & Smith, D. E. (2000). Assessment center exercise factors represent cross-situational specificity, not method bias. Human Performance, 13, 323–353.10.1207/S15327043HUP1304_1Google Scholar
Lievens, F. (2008). What does exercise-based assessment really mean? Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 112–115.10.1111/j.1754-9434.2007.00020.xGoogle Scholar
Lievens, F., Chasteen, C. S., Day, E. A., & Christiansen, N. D. (2006). Large-scale investigation of the role of trait activation theory for understanding assessment center convergent and discriminant validity. Journal of Applied Psychology, 91, 247–258.10.1037/0021-9010.91.2.247Google Scholar
Lievens, F., & Conway, J. M. (2001). Dimensions and exercise variance in assessment center scores: A large-scale evaluation of multitrait-multimethod studies. Journal of Applied Psychology, 86, 1202–1222.10.1037/0021-9010.86.6.1202Google Scholar
Lievens, F., & Klimoski, R. J. (2001). Understanding the assessment centre process: Where are we now? International Review of Industrial and Organizational Psychology, 16, 246–286.Google Scholar
Lovler, B., Rose, M., & Wesley, S. (2002, April). Finding assessment center construct validity: Try behaviors instead of dimensions. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Toronto, Ontario, Canada.Google Scholar
Lowry, P. E. (1995). The assessment center process: Assessing leadership in the public sector. Public Personnel Management, 24, 443–450.10.1177/009102609502400403Google Scholar
Lowry, P. E. (1997). The assessment center process: New directions. Journal of Social Behavior and Personality, 12(5), 53–62.Google Scholar
Melchers, K. G., & König, C. J. (2008). It is not yet time to dismiss dimensions in assessment centers. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 125–127.10.1111/j.1754-9434.2007.00023.xGoogle Scholar
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.10.1037/0003-066X.50.9.741Google Scholar
Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81–97.10.1037/h0043158Google Scholar
Mischel, W. (1999). Personality coherence and dispositions in a cognitive-affective personality (CAPS) approach. In Cervone, D. & Shoda, Y. (Eds.), The coherence of personality: Social-cognitive bases of consistency, variability, and organization (pp. 37–60). New York: Guilford.Google Scholar
Moses, J. L. (1977). The assessment center method. In Moses, J. L. & Byham, W. C. (Eds.), Applying the assessment center method (pp. 3–11). New York: Pergamon.10.1016/B978-0-08-019581-0.50006-2Google Scholar
Moses, J. L. (2008). Assessment centers work, but for different reasons. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 134–136.10.1111/j.1754-9434.2007.00026.xGoogle Scholar
Moses, J. L., & Byham, W. C. (Eds.). (1977). Applying the assessment center method. New York: Pergamon.Google Scholar
Murphy, K. R. (in press). Explaining the weak relationship between job performance and ratings of job performance. Industrial and Organizational Psychology: Perspectives on Science and Practice.Google Scholar
Neidig, R. D., & Neidig, P. J. (1984). Multiple assessment center exercises and job relatedness. Journal of Applied Psychology, 69, 182–186.10.1037/0021-9010.69.1.182Google Scholar
Rupp, D. E., Thornton, G. C. III, & Gibbons, A. M. (2008). The construct validity of the assessment center method and usefulness of dimensions as focal constructs. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 116–120.10.1111/j.1754-9434.2007.00021.xGoogle Scholar
Sackett, P. R., & Dreher, G. F. (1982). Constructs and assessment center dimensions: Some troubling findings. Journal of Applied Psychology, 67, 401–410.10.1037/0021-9010.67.4.401Google Scholar
Sackett, P. R., & Dreher, G. F. (1984). Situation specificity of behavior and assessment center validation strategies: A rejoinder to Neidig and Neidig. Journal of Applied Psychology, 69, 187–190.10.1037/0021-9010.69.1.187Google Scholar
Schleicher, D. J., Day, D. V., Mayes, B. T., & Riggio, R. E. (2002). A new frame for frame-of-reference training: Enhancing the construct validity of assessment centers. Journal of Applied Psychology, 87, 735–746.10.1037/0021-9010.87.4.735Google Scholar
Schuler, H. (2008). Improving assessment centers by the trimodal concept of personnel assessment. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 128–130.10.1111/j.1754-9434.2007.00024.xGoogle Scholar
Shoda, Y. (2003). Studying persons in order to understand situations; studying situations in order to understand persons. In Sansone, C., Morf, C., & Panter, A. (Eds.), Handbook of methods in social psychology (pp. 128–153). Thousand Oaks, CA: Sage.Google Scholar
Shoda, Y., Mischel, W., & Wright, J. C. (1993). The role of situational demands and cognitive competencies in behavior organization and personality coherence. Journal of Personality and Social Psychology, 65, 1023–1035.10.1037/0022-3514.65.5.1023Google Scholar
Tett, R. P., & Burnett, D. D. (2003). A personality trait-based interactionist model of job performance. Journal of Applied Psychology, 88, 500–517.10.1037/0021-9010.88.3.500Google Scholar
Thoresen, C. J., Lance, C. E., & Thoresen, J. D. (2007, April). Exercises, not dimensions, are the currency of assessment centers. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, New York, NY.Google Scholar
Thornton, G. C., , III. (1992). Assessment centers in human resource management. Reading, MA: Addison-Wesley.Google Scholar
Thornton, G. C. III., & Byham, W. C. (1982). Assessment centers and managerial performance. New York: Academic.Google Scholar
Tubré, T., Arthur, W. Jr., & Bennett, W. Jr. (2006). General models of job performance: Theory and practice. In Bennett, W. Jr., Lance, C. E., & Woehr, D. J. (Eds.), Performance measurement: Current perspectives and future challenges (pp. 175–203). Mahwah, NJ: Erlbaum.Google Scholar
Vandenberg, R. J. (2006). Statistical and methodological myths and urban legends: Where, pray tell, did they get this idea? Organizational Research Methods, 9, 194–201.10.1177/1094428105285506Google Scholar
Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108–131.10.1037/0021-9010.90.1.108Google Scholar
Woehr, D. J., & Arthur, W. Jr. (2003). The construct-related validity of assessment center ratings: A review and meta-analysis of the role of methodological factors. Journal of Management, 29, 231–258.10.1177/014920630302900206Google Scholar