Skip to main content Accessibility help
×
Home

Information:

  • Access

Actions:

      • Send article to Kindle

        To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Don't Forget To Properly Use Your Signal: Driving Down New Roads to Selection Decisions
        Available formats
        ×

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Don't Forget To Properly Use Your Signal: Driving Down New Roads to Selection Decisions
        Available formats
        ×

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Don't Forget To Properly Use Your Signal: Driving Down New Roads to Selection Decisions
        Available formats
        ×
Export citation

Extract

Chamorro-Premuzic, Winsborough, Sherman, and Hogan (2016) note that new talent signals recently adopted by organizations are related to older selection and assessment methods. Drawing this connection between old and new technologies is helpful; however, viewing new technology as either shiny new objects or a brave new world creates a false dichotomy. Recent technology-enhanced human resources (HR) processes like the widespread use of gamified practices and video-recorded interviewing are not just fads or the beginning of a transformation in HR but rather natural evolutions of methods that differ across specific dimensions that can be identified and measured. It is important to view these recent advances as extensions of the existing methods. That is, we need to focus on how these new methods are different and not on that they are different.

Chamorro-Premuzic, Winsborough, Sherman, and Hogan (2016) note that new talent signals recently adopted by organizations are related to older selection and assessment methods. Drawing this connection between old and new technologies is helpful; however, viewing new technology as either shiny new objects or a brave new world creates a false dichotomy. Recent technology-enhanced human resources (HR) processes like the widespread use of gamified practices and video-recorded interviewing are not just fads or the beginning of a transformation in HR but rather natural evolutions of methods that differ across specific dimensions that can be identified and measured. It is important to view these recent advances as extensions of the existing methods. That is, we need to focus on how these new methods are different and not on that they are different.

Understanding how these methods for identifying talent differ is important because they have measurable psychological effects on users. For example, we can compare video-recorded interviews with in-person interviews or synchronous online interviews, but such comparisons need to be articulated in terms of specific differences between the administration media. In this reply, we provide a path forward for research on new selection tools and advise caution to not forego good scientific practice when utilizing these tools.

A Path Forward

In order to better understand how technological advances can affect organizational processes like personnel selection, we need to develop a better understanding of technology itself. Many of industrial–organizational (I-O) psychology's past and current discussions on technology have focused on assessing how particular hardware or software innovations change current practice. A number of studies have been conducted in which two technologies are pitted against each other to draw conclusions about their usage. Although these studies have each added value to their respective fields of study, they do not necessarily help us to determine how the next major technological innovation will affect organizations. As technology moves at an ever increasing rate, we need to move beyond examining individual technologies and start building frameworks grounded in psychological theory to understand the ways technologies can vary from one another.

This call for the development of technological taxonomies and frameworks is not new. A number of researchers have developed frameworks to describe ways that technology can affect communication (Daft, Lengel, & Ven, 1986; Maruping & Agarwal, 2004) and assessments in personnel selection (Leeson, 2006; Potosky, 2008). For example, Potosky's (2008) framework suggests that assessments administered through a medium are affected by the medium's transparency, social bandwidth, interactivity, and surveillance. Propositions are also supplied for how each of these characteristics should affect assessment. These frameworks have existed for several years; however, they are seldom empirically tested in I-O psychology research. Without empirical testing, these frameworks remain in a state of limbo; that is, they aim to help us understand technology, but their lack of empirical support means that we cannot be sure of their propositions. These frameworks can be of immense use, but they need to first be tested. We currently lack the evidence to say whether extant taxonomies or frameworks are sufficient or whether they require expansions or modifications. Moreover, the lack of integration of these frameworks in the I-O psychology literature helps to reinforce the common approach of making individual comparisons of technology and keeps us further away from developing an understanding of future innovations. By testing and refining technological frameworks, we can become closer as a field to being able to make informed predictions as to how new forms of technology will affect organizations, without the need for comparing the “old” with the “new” methods.

Chamorro-Premuzic et al. use video-recorded responses to structured interviews as an example of a new version of traditional assessment methods. Although they are correct that this is a new technology, it is not wholly unlike extant interview methods. Rather than examining video-recorded interviews and synchronous online interviewing as two separate phenomena, we should examine how they differ. That is, we can apply frameworks like Potosky's to make specific predictions as to how they might differ. This will allow us to understand the next great innovation in interviewing, whatever that may be. Better yet, we will be in a position to offer advice as to what that next step should be.

Chamorro-Premuzic et al. also address gamification, which, like video-recorded interviewing, is best thought of as an extension of older selection methods. Organizations are drawn to gamified selection tools by a belief that these tools will be more engaging for candidates than traditional selection tools. While much research has been devoted to understanding gamified training and subsequent learning outcomes (Landers & Armstrong, 2014; Landers & Callan, 2011; Orvis, Horn, & Belanich, 2008; Wilson et al., 2008), virtually no research has focused on gamified selection. The dearth of published research in this area should be viewed as a major hindrance to our ability to fully utilize these tools and be confident in decisions made using them. All is not lost, however; I-O psychologists are experts in understanding motivation and how goals are affected by feedback. We can draw from what we know about traditional selection tools and motivation and apply it to these new approaches to develop a full understanding of how gamified selection tools might affect candidates in different ways.

In gamified selection, there are aspects of the technology that affect candidates in unintended but ultimately scientifically predictable ways. Cognitive load theory suggests that tasks performed in unfamiliar domains are more likely to tax working memory (Kalyuga, Chandler, & Sweller, 2001). That is, when operating in unfamiliar domains, candidates are likely to experience inhibited performance due to increases in extraneous cognitive load (Chandler & Sweller, 1991). In line with cognitive load theory, Orvis et al. (2008) found prior videogame experience resulted in higher motivation and better performance in instructional videogames. Candidates with less videogame experience are likely to be at a disadvantage, but research is needed to investigate these issues in this new context. Recent research on video interviews and technology-mediated communication has further shown how new selection methods might differ from old ones as a product of the unique aspects of the technology that are used. Horn and Behrend (2016) found that the presence of the picture-in-picture window, which allows users to see themselves during a video (Skype) interview resulted in higher cognitive load than when the window was not present. The picture-in-picture window acts as an attentional distractor, which increases cognitive load on complex tasks (Baron, Moore, & Sanders, 1978). Further, communication through video has been shown to elicit higher levels of cognitive load than audio (Hinds, 1999) or face to face (Ferran & Watts, 2008). These findings elucidate the need to better understand the impact of particular aspects of selection tools.

Context and Scenery

As we navigate these new roads to selection decisions we must not forget that changes in scenery affect applicants. The convenience of technology often results in organizations becoming lax when it comes to the degree of structure implemented in assessments. As technology provides greater levels of autonomy to candidates, there are more opportunities to introduce error into assessment scores. Gamified tests and video interviews allow users to choose the environment in which they perform. When choosing where to take an assessment one candidate might choose to lounge in a cool quiet room, whereas another candidate who does not have this luxury may instead opt for the precarious stool in the crowded coffee shop down the street. These choices about context are not without consequences, as we have learned countless times from varying angles over the years, such as in research on situational strength (e.g., Meyer, Dalal, & Bonaccio, 2012) and trait activation (e.g., Tett & Burnett, 2003). Each context could introduce idiosyncratic elements that affect performance in positive or negative ways. Temperature (Halali, Meiran, & Shalev, 2016) and office design (Seddigh et al., 2015) are a few examples of contextual factors shown to affect performance on demanding cognitive tasks. Researchers are trained to take environment into account when designing experimental studies, but unfortunately this same care is not always taken in practice. If we are to be confident in decisions made using new tools, we must remember to account for the consequences of the autonomy that comes with them.

Conclusion

Chamorro-Premuzic et al. have successfully given a brief overview of new selection tools. Our mission was to provide a path forward and a way of thinking when researching these tools. If we continue to look at each new technology as a wholly new phenomenon, we will be repeatedly asking the same question—How does new x compare to old y?—without really learning anything in the process. If we instead understand exactly how technologically enhanced selection methods differ from one another, we can then learn more about how the next innovation will affect organizational processes and in turn make more informed decisions. This will allow us to generate an understanding of technology that is more stable and that will be more robust to future technology changes. Finally, we acknowledge that it is impossible for research to keep perfect pace with practice, but we should aim to make that gap as narrow as possible. By improving our understanding of the ways that technologies can affect users, researchers may be more able to keep up with their fellow practitioners in terms of studying and using new innovations. Similarly, practitioners can use this enhanced understanding to take additional steps to control for contextual factors during assessment delivery.

References

Baron, R. S., Moore, D., & Sanders, G. S. (1978). Distraction as a source of drive in social facilitation research. Journal of Personality and Social Psychology, 36 (8), 816824. http://doi.org/10.1037/0022-3514.36.8.816
Chamorro-Premuzic, T., Winsborough, D., Sherman, R. A., & Hogan, R. (2016). New talent signals: Shiny new objects or a brave new world? Industrial and Organizational Psychology: Perspectives on Science and Practice, 9 (3), 621640.
Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8 (4), 293332. http://doi.org/10.1207/s1532690xci0804_2
Daft, R. L., Lengel, R. H., & Ven, V. De. (1986). Organizational information requirements, media richness and structural design. Management Science, 32 (5), 554571. http://doi.org/10.1287/mnsc.32.5.554
Ferran, C., & Watts, S. (2008). Videoconferencing in the field: A heuristic processing model. Management Science, 54 (9), 15651578. http://doi.org/10.1287/mnsc.1080.0879
Halali, E., Meiran, N., & Shalev, I. (2016). Keep it cool: Temperature priming effect on cognitive control. Psychological Research. Advance online publication. http://doi.org/10.1007/s00426-016-0753-6
Hinds, P. J. (1999). The cognitive and interpersonal costs of video. Media Psychology, 1 (4), 283311. http://doi.org/10.1207/s1532785xmep0104
Horn, R. G., & Behrend, T. S. (2016, April). Video killed the interview star: Does picture-in-picture affect interview performance? Poster presented at the 31st Annual Conference of the Society for Industrial and Organizational Psychology, Anaheim, CA.
Kalyuga, S., Chandler, P., & Sweller, J. (2001). Learner experience and efficiency of instructional guidance. Educational Psychology, 21 (1), 523. http://doi.org/10.1080/01443410124681
Landers, R. N., & Armstrong, M. B. (2014). Enhancing instructional outcomes with gamification: An empirical test of the technology-enhanced training effectiveness model. Computers in Human Behavior. Advance online publication. http://doi.org/10.1016/j.chb.2015.07.031
Landers, R. N., & Callan, R. C. (2011). Casual social games as serious games: The psychology of gamification in undergraduate education and employee training. In Ma, M., Oikonomou, A., & Jain, L. C. (Eds.), Serious games and edutainment applications (pp. 399423). New York, NY: Springer-Verlag. http://doi.org/10.1007/978-1-4471-2161-9
Leeson, H. V. (2006). The mode effect: A literature review of human and technological issues in computerized testing. International Journal of Testing, 6 (1), 124. http://doi.org/10.1207/s15327574ijt0601_1
Maruping, L. M., & Agarwal, R. (2004). Managing team interpersonal processes through technology: A task-technology fit perspective. Journal of Applied Psychology, 89 (6), 975990. http://doi.org/10.1037/0021-9010.89.6.975
Meyer, R. D., Dalal, R. S., & Bonaccio, S. (2012). A meta-analytic investigation into the moderating effects of situational strength on the conscientiousness-performance relationship. Journal of Organizational Behavior, 33 (1), 148150. http://doi.org/10.1002/job.786
Orvis, K. A., Horn, D. B., & Belanich, J. (2008). The roles of task difficulty and prior videogame experience on performance and motivation in instructional videogames. Computers in Human Behavior, 24 (5), 24152433. http://doi.org/10.1016/j.chb.2008.02.016
Potosky, D. (2008). A conceptual framework for the role of the administration medium in the personnel assessment process. Academy of Management Review, 33 (3), 629648. http://doi.org/10.5465/AMR.2008.32465704
Seddigh, A., Stenfors, C., Berntsson, E., Bååth, R., Sikström, S., & Westerlund, H. (2015). The association between office design and performance on demanding cognitive tasks. Journal of Environmental Psychology, 42, 172181. http://doi.org/10.1016/j.jenvp.2015.05.001
Tett, R. P., & Burnett, D. D. (2003). A personality trait-based interactionist model of job performance. Journal of Applied Psychology, 88 (3), 500517. http://doi.org/10.1037/0021-9010.88.3.500
Wilson, K. A., Bedwell, W. L., Lazzara, E. H., Salas, E., Burke, C. S., Estock, J. L., . . . Conkey, C. (2008). Relationships between game attributes and learning outcomes: Review and research proposals. Simulation & Gaming, 40 (2), 217266. http://doi.org/10.1177/1046878108321866