Skip to main content Accessibility help
×
Home

Where Have We Been, How Did We Get There, and Where Shall We Go?

  • Charles E. Lance (a1)
  • In response to commentaries on:

Abstract

Commentators expressed a wide variety of views on my evaluation of the state of assessment center (AC) research and practice. In this response, I first trace the evolution of the construct validity paradox “urban legend.” Next, I consider the commentators’ comments as they relate to (a) my recommendation to abandon dimension-based ACs in lieu of task- or role-based structures (b) my recommendation to discontinue design fix attempts toward making ACs conform to multitrait–multimethod construct validity criteria, and (c) considerations of construct validity and validation evidence. Finally, I offer some directions for future AC research and practice.

Copyright

Corresponding author

E-mail: clance@uga.edu, Address: Department of Psychology, The University of Georgia, Athens, GA 30602-3013

References

Hide All
Arthur, W. Jr., Day, E. A., & Woehr, D. J. (2008). Mend it, don’t end it: An alternate view of assessment center construct-related validity evidence. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 105111.
Arthur, W. Jr., Woehr, D. J., & Maldegan, R. (2000). Convergent and discriminant validity of assessment center dimensions: A conceptual and empirical re-examination of the assessment center construct-related validity paradox. Journal of Management, 26, 813835.
Bergman, A., & Bergman, M. (1973). The way we were [Recorded by B. Streisand]. On The way we were [CD]. Tokyo: Sony. (2002).
Binning, J. F., & Barrett, G. V. (1988). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74, 478494.
Borsboom, D., Mellenbergh, G. J., & Van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 10611071.
Bowler, M. C., & Woehr, D. J. (2006). A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings. Journal of Applied Psychology, 91, 11141124.
Brannick, M. T. (2008). Back to basics of test construction and scoring. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 131133.
Bray, D. W., Campbell, R. J., & Grant, D. L. (1974). Formative years in business: A long-term AT&T study of managerial lives. New York: Wiley.
Bray, D. W., & Grant, D. L. (1966). The assessment center in the measurement of potential for business management. Psychological Monographs, 80 (17, Whole No. 625), 127.
Campbell, J. P. (1990). Modeling the performance prediction problem in industrial and organizational psychology. In Dunnette, M. D. & Hough, L. M. (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 1, pp. 687732). Palo Alto, CA: Consulting Psychologists Press.
Campbell, J. P., McHenry, J. J., & Wise, L. L. (1990). Modeling job performance in a population of jobs. Personnel Psychology, 43, 313333.
Cervone, D., & Shoda, Y. (1999). Social-cognitive theories and the coherence of personality. In Cervone, D. & Shoda, Y. (Eds.), The coherence of personality: Social-cognitive bases of personality consistency, variability, and organization (pp. 333). New York: Guilford.
Connelly, B. S., Ones, D. S., Ramesh, A., & Goff, M. (2008). A pragmatic view of assessment center exercises and dimensions. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 121124.
Cowan, N. (2000). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87114.
Dunnette, M. D. (1971). The assessment of managerial talent. In McReynolds, P. (Ed.), Advances in psychological assessment (Vol. 2, pp. 79108). Palo Alto, CA: Science and Behavior Books.
Ekehammer, B. (1974). Interactionism in personality from a historical perspective. Psychological Bulletin, 81, 10261048.
Howard, A. (1997). A reassessment of assessment centers: Challenges for the 21st century. Journal of Social Behavior and Personality, 12(5), 1352.
Howard, A. (2008). Making assessment centers work the way they’re supposed to. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 98104.
Jackson, D. J. R., Barney, A. R., Stillman, J. A., & Kirkley, B. (2007). When traits are behaviors: The relationship between behavioral responses and trait-based overall assessment center ratings. Human Performance, 20, 415432.
Jackson, D. J. R., Stillman, J. A., & Atkins, S. G. (2005). Rating tasks versus dimensions in assessment centers: A psychometric comparison. Human Performance, 18, 213241.
Jones, R. G., & Klimoski, R. J. (2008). Narrow standards for efficacy and the research playground: Why either–or conclusions do not help. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 137139.
Joyce, L. W., Thayer, P. W., & Pond, S. B. III (1994). Managerial functions: An alternative to traditional assessment center dimensions? Personnel Psychology, 47, 109121.
Kelly, T. L. (1927). Interpretations of educational measurements. New York: Macmillan.
Lance, C. E., Baranik, L. E., Lau, A. R., & Scharlau, E. A. (in press). If it ain’t trait it must be method: (Mis) application of the multitrait-multimethod methodology in organizational research. In Lance, C. E. & Vandenberg, R. J. (Eds.), Statistical and methodological myths and urban legends: Received doctrine, verity, and fable in organizational and social research. New York: Routledge.
Lance, C. E., Foster, M. R., Drollinger, S. M., Sorensen, K. L., Gentry, W. A., & Nemeth, Y. M. (2007, April). A comparison of task-based versus dimension-based scoring procedures in an operational assessment center (AC). Paper presented at the meeting of the Society for Industrial and Organizational Psychology, New York, NY.
Lance, C. E., Foster, M. R., Gentry, W. A., & Thoresen, J. D. (2004). Assessor cognitive processes in an operational assessment center. Journal of Applied Psychology, 89, 2235.
Lance, C. E., Foster, M. R., Nemeth, Y. M., Gentry, W. A., & Drollinger, S. (2007). Extending the nomological network of assessment center construct validity: Prediction of cross-situationally consistent and specific aspects of assessment center performance. Human Performance, 20, 345362.
Lance, C. E., Lambert, T. A., Gewin, A. G., Lievens, F., & Conway, J. M. (2004). Revised estimates of dimension and exercise variance components in assessment center post-exercise dimension ratings. Journal of Applied Psychology, 89, 377385.
Lance, C. E., Newbolt, W. H., Gatewood, R. D., Foster, M. R., French, N., & Smith, D. E. (2000). Assessment center exercise factors represent cross-situational specificity, not method bias. Human Performance, 13, 323353.
Lievens, F. (2008). What does exercise-based assessment really mean? Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 112115.
Lievens, F., Chasteen, C. S., Day, E. A., & Christiansen, N. D. (2006). Large-scale investigation of the role of trait activation theory for understanding assessment center convergent and discriminant validity. Journal of Applied Psychology, 91, 247258.
Lievens, F., & Conway, J. M. (2001). Dimensions and exercise variance in assessment center scores: A large-scale evaluation of multitrait-multimethod studies. Journal of Applied Psychology, 86, 12021222.
Lievens, F., & Klimoski, R. J. (2001). Understanding the assessment centre process: Where are we now? International Review of Industrial and Organizational Psychology, 16, 246286.
Lovler, B., Rose, M., & Wesley, S. (2002, April). Finding assessment center construct validity: Try behaviors instead of dimensions. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Toronto, Ontario, Canada.
Lowry, P. E. (1995). The assessment center process: Assessing leadership in the public sector. Public Personnel Management, 24, 443450.
Lowry, P. E. (1997). The assessment center process: New directions. Journal of Social Behavior and Personality, 12(5), 5362.
Melchers, K. G., & König, C. J. (2008). It is not yet time to dismiss dimensions in assessment centers. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 125127.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741749.
Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 8197.
Mischel, W. (1999). Personality coherence and dispositions in a cognitive-affective personality (CAPS) approach. In Cervone, D. & Shoda, Y. (Eds.), The coherence of personality: Social-cognitive bases of consistency, variability, and organization (pp. 3760). New York: Guilford.
Moses, J. L. (1977). The assessment center method. In Moses, J. L. & Byham, W. C. (Eds.), Applying the assessment center method (pp. 311). New York: Pergamon.
Moses, J. L. (2008). Assessment centers work, but for different reasons. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 134136.
Moses, J. L., & Byham, W. C. (Eds.). (1977). Applying the assessment center method. New York: Pergamon.
Murphy, K. R. (in press). Explaining the weak relationship between job performance and ratings of job performance. Industrial and Organizational Psychology: Perspectives on Science and Practice.
Neidig, R. D., & Neidig, P. J. (1984). Multiple assessment center exercises and job relatedness. Journal of Applied Psychology, 69, 182186.
Rupp, D. E., Thornton, G. C. III, & Gibbons, A. M. (2008). The construct validity of the assessment center method and usefulness of dimensions as focal constructs. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 116120.
Sackett, P. R., & Dreher, G. F. (1982). Constructs and assessment center dimensions: Some troubling findings. Journal of Applied Psychology, 67, 401410.
Sackett, P. R., & Dreher, G. F. (1984). Situation specificity of behavior and assessment center validation strategies: A rejoinder to Neidig and Neidig. Journal of Applied Psychology, 69, 187190.
Schleicher, D. J., Day, D. V., Mayes, B. T., & Riggio, R. E. (2002). A new frame for frame-of-reference training: Enhancing the construct validity of assessment centers. Journal of Applied Psychology, 87, 735746.
Schuler, H. (2008). Improving assessment centers by the trimodal concept of personnel assessment. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 128130.
Shoda, Y. (2003). Studying persons in order to understand situations; studying situations in order to understand persons. In Sansone, C., Morf, C., & Panter, A. (Eds.), Handbook of methods in social psychology (pp. 128153). Thousand Oaks, CA: Sage.
Shoda, Y., Mischel, W., & Wright, J. C. (1993). The role of situational demands and cognitive competencies in behavior organization and personality coherence. Journal of Personality and Social Psychology, 65, 10231035.
Tett, R. P., & Burnett, D. D. (2003). A personality trait-based interactionist model of job performance. Journal of Applied Psychology, 88, 500517.
Thoresen, C. J., Lance, C. E., & Thoresen, J. D. (2007, April). Exercises, not dimensions, are the currency of assessment centers. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, New York, NY.
Thornton, G. C., , III. (1992). Assessment centers in human resource management. Reading, MA: Addison-Wesley.
Thornton, G. C. III., & Byham, W. C. (1982). Assessment centers and managerial performance. New York: Academic.
Tubré, T., Arthur, W. Jr., & Bennett, W. Jr. (2006). General models of job performance: Theory and practice. In Bennett, W. Jr., Lance, C. E., & Woehr, D. J. (Eds.), Performance measurement: Current perspectives and future challenges (pp. 175203). Mahwah, NJ: Erlbaum.
Vandenberg, R. J. (2006). Statistical and methodological myths and urban legends: Where, pray tell, did they get this idea? Organizational Research Methods, 9, 194201.
Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108131.
Woehr, D. J., & Arthur, W. Jr. (2003). The construct-related validity of assessment center ratings: A review and meta-analysis of the role of methodological factors. Journal of Management, 29, 231258.

Related content

Powered by UNSILO

Where Have We Been, How Did We Get There, and Where Shall We Go?

  • Charles E. Lance (a1)
  • In response to commentaries on:

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed.