Sample & Procedure
The study was conducted in Greece during the last months of 2017, attracting participants via the authors’ university career office, along with post-graduate and final-year undergraduate students or recent graduates. We contacted final-year undergraduate students, graduate students or recent graduates to participate in a survey about a selection method, as these students were approaching graduation and were likely to search for employment soon (e.g., van Iddekinge, Lanivich, Roth, & Junco, Reference van Iddekinge, Lanivich, Roth and Junco2016).
The data collection took place in two phases. In the first phase, participants were invited to complete the self-reported measures of cognitive ability, personality, performance measures and OCB. Three to four weeks after completion, participating individuals of the first phase were invited to play the game-based assessment. 193 participants took part in the first phase and 120 of them participated in the second phase, as well, a response rate of 62%. The majority of them were females (64%) with a mean age of 26 years. As far as their education level is concerned, 46% of the participants were final year undergraduates, 15% were post-graduate students, another 15% were university graduates and 24% had already acquired a post-graduate degree. Most of them (55%) were currently employed, working in entry-level (57.5%) or middle-level positions (27.5%).
Cognitive ability. This was measured with items taken from the International Cognitive Ability Resource (ICAR) (2014),Footnote 1. ICAR is a public-domain and open-source tool created by Condon and Revelle (Reference Condon and Revelle2014), aiming to provide a large and dynamic bank of cognitive ability measures for use in a wide variety of applications, including research. The test includes four item types: Three-Dimensional Rotations, Letter and Number Series, Matrix Reasoning, and Verbal Reasoning. We used the 11 Matrix Reasoning items, which contain stimuli similar to those used in Raven’s Progressive Matrices, and which is also more closely related to abstract reasoning. “The stimuli are 3x3 arrays of geometric shapes with one of the nine shapes missing. Participants are instructed to identify which of six geometric shapes presented as response choices will best complete the stimuli” (ICAR, 2014, p. 2).Footnote 2 It is worth noting that the correct answer is only one, whereas the options “None of the above” and “Do not know” are also available. An overall score is calculated, with high scores indicating higher levels of cognitive abilityFootnote 3.
Personality. Participants completed the 50 items International Personality Item Pool (IPIP; Goldberg et al., Reference Goldberg, Johnson, Eber, Hogan, Ashton, Cloninger and Gough2006) to assess the Five-Factor model of personality. Each scale consisted of 10 items. Standard IPIP instructions were presented to participants, who responded on a 5-point Likert-type scale ranging from 1 (inaccurate) to 5 (accurate). Research has reported good internal consistencies for IPIP factors (see, for example, Lim & Ployhart, Reference Lim and Ployhart2006). In our study, reliability estimates were .81 for conscientiousness, .83 for emotional stability, .83 for extroversion, .79 for agreeableness, and .75 for openness to experience.
Performance measures. Overall job performance was self-evaluated by working individuals only using a measure used by Nikolaou and Robertson (Reference Nikolaou and Robertson2001). It consists of six items where the individual has to indicate whether she/he agrees or disagrees with the behavior described in a five-point scale ranging from 1 (strongly disagree) to 5 (strongly agree). An overall job performance score was calculated by averaging the scores of the six items eliciting internal consistency reliability of .91. Example items include “Achieve the objectives of the job” and “Demonstrates expertise in all aspects of the job”. We also asked participants to indicate their GPA from their first degree in order to use it as an alternative to job performance for non-working individuals. The range of the grading system in Greek public universities is 0.00–10.00 (Excellent = 8.50–10.00, Very Good = 6.5–8.49, Good = 5.00 –6.49, and Fail = 0.00–4.59). The GPA reported by participants was the average grade awarded for the duration of their bachelor studies.
Organizational Citizenship Behavior (OCB). OCBs were self-evaluated by working individuals only using a measure developed by Smith et al. (Reference Smith, Organ and Near1983). It consists of 16 items where the individual has to indicate whether she/he agrees or disagrees with the behavior described in a five-point scale ranging from 1 (strongly disagree) to 5 (strongly agree). The original scale measures two subscales; altruism and generalized compliance. However, for the purposes of the current study we only used the overall OCB score eliciting internal consistency reliability of .70. Example items include “I help other employees with their work when they have been absent” and “I exhibit punctuality in arriving at work on time in the morning and after lunch breaks”.
Soft skills. We used a Game-Based Assessment (GBA) developed by OwiwiFootnote 4 in order to measure the four soft skills evaluated by the game, namely resilience, adaptability, flexibility and decision-making. The four skills are evaluated following a SJT methodology converted into an on-line game environment, with fictional characters. The Owiwi game has demonstrated satisfactory psychometric elements and increased equivalence with the originally developed SJT measuring the four soft skills (Georgiou, Nikolaou, & Gouras, Reference Georgiou, Nikolaou and Gouras2017). Resilience is defined as “the developable capacity to rebound or bounce back from adversity, conflict, and failure or even positive events, progress, and increased responsibility” (Luthans, Reference Luthans2002, p. 702), “Αdaptability is related to change and how people deal with it; that is to say, people’s adjustment to changing environments” (Hamtiaux et al., Reference Hamtiaux, Houssemand and Vrignaud2013, p. 130). Flexibility is defined as the demonstration of “adaptable as opposed to routine behaviors; it is the extent to which employees possess a broad repertoire of behavioral scripts that can be adapted to situation-specific demands” (Bhattacharya et al., Reference Bhattacharya, Gibson and Doty2005, p. 624) and finally decision-making is defined as an intellectual process leading to a response to circumstances through selection among alternatives (Nelson, Reference Nelson1984). Individualized feedback is provided to all participants upon completion of the game.
Table 1 presents the inter-correlation matrix of the study’s variables. An interesting pattern we observe in the inter-correlation matrix, is that the cognitive ability measure is not associated with any of the scales measured here. Also, the self-reported job performance measure is correlated significantly with conscientiousness, emotional stability and openness to experience for the five-factor model of personality. Moreover, the OCB measure is associated with agreeableness, similarly to past research on the relationships between agreeableness and OCB, but not with conscientiousness. Finally, the soft skills assessed by the game-based assessment, which is the main focus of the current study, are not correlated with any of the criterion measures, with the exception of the positive correlation between GPA and decision making, rejecting thus H 1 and H 3 and only partially confirming H 2.
Table 1. Inter-Correlation Matrix of Study’s Variables (N = 63–120)
Next, we proceed with the examination of our research hypotheses. Our main focus in this study is the suitability of the game-based assessment as a selection tool, above and beyond the well-established effect of cognitive ability and personality, especially conscientiousness. Our first three hypotheses deal with the association between game-based assessment and the three performance criteria. In order to explore these hypotheses we executed three separated multiple regression analyses for each one of the three criterion measures. The results of these analyses are presented in Table 2.
Table 2. Hierarchical Regression Analysis of the GBA on the Three Criterion Measures
The results of the regression analyses show that flexibility and decision-making are positively associated with self-reported job performance and GPA respectively. The block of the four skills predict 13%, 7% and 10% of the total variance in job performance, OCB and GPA respectively. Therefore, H 1 and H 2 are partially confirmed, whereas H 3 is rejected. Subsequently, we explored the incremental validity of the game-based assessment. In order to explore H 4-H 6 we conducted a number of hierarchical regression analyses, controlling for the effect of cognitive ability and the five-factor model of personality. The results of these analyses are presented in Table 3.
Table 3. Hierarchical Regression Analysis of the GBA on the Three Criterion Measures controlling for Cognitive Ability and Personality
The results of these analyses demonstrate that the soft skills measured by the game-based assessment do not predict additional variance in either job performance or OCBs for the working individuals of our sample, above the effect of cognitive ability and personality rejecting thus H 4 and H 6. However, they seem to have an important effect on GPA. More specifically, both as a group and separately (adaptability and decision making) demonstrate a statistical significant relationship with GPA, above and beyond the effect of cognitive ability and personality. These results establish the usefulness of game-based assessments in predicting educational attainment, as measured by the GPA, both as a group and individually in the case of adaptability and decision making.
Our study explores the effectiveness of a game-based assessment in employee selection. Extending previous research on Work/Organizational Psychology and traditional selection methods, we introduce a game-based assessment designed to measure candidates’ soft skills (e.g., adaptability, flexibility, decision-making) that is found to be associated with self-reported measures of performance. Our study contributes to employee selection research, providing some support to the use of gamification in soft skills assessments and their ability to predict performance in work and academic settings. For example, a game-based assessment measuring soft skills, such as decision-making and flexibility, can predict test-takers’ self-reported job performance and GPA. By incorporating game elements into assessments that do not use self-reported measures, but assess behavioral intentions, test-takers’ attractiveness and engagement into the assessment might be enhanced, while it might be more difficult for them to understand what is being assessed and what the correct answer is (Armstrong et al., Reference Armstrong, Landers, Collmus, Davis and Gangadharbatla2016; Fetzer et al., Reference Fetzer, McNamara, Geimer, Goldstein, Pulakos, Passmore and Semedo2017). As such, the use of game elements and designs might improve the validity of assessments.
Moreover, Armstrong et al. (Reference Armstrong, Landers, Collmus, Davis and Gangadharbatla2016) suggested that game-based assessments, such as gamified simulations, might be employed to assess important predictor constructs like learning agility in employee selection settings where survey methodology may not be adequate. Along these lines, our study extends research on traditional selection methods, exploring the incremental validity of a game-based assessment assessing soft skills. Game-based assessments measuring soft skills, such as adaptability and decision making, can predict academic performance (e.g., GPA), above and beyond traditional selection methods (e.g., cognitive ability and personality tests). However, the soft skills measured by the game-based assessment do not predict additional variance in either job performance or OCBs, above the effect of cognitive ability and personality.
To sum up, both personality and intelligent tests have been extensively tested in academic contexts and their validity in predicting GPA has been established. The emergence of internet and technology as well as the familiarity of new generations with games are likely to reflect an increasing interest in the validity of game-based assessments in predicting academic performance beyond traditional selection methods. The additive value of using a game-based assessment measuring adaptability and decision making, both as a group and individually, in predicting GPA beyond personality (e.g., FFM) and cognitive ability tests (e.g., ICAR), has been established.
Our results are of interest to researchers and practitioners of Work/Organizational Psychology interested in the prediction of work and academic performance, in that they support the incremental validity of a game-based assessment over and above traditional selection methods. They contribute to empirical unknowns about the psychometrics properties and effectiveness of the use of game-based assessments in employee selection.
Game-based assessments might be used as a supplement or replacement tool to traditional selection methods as they add to the prediction of performance of candidates or students. However, it is of high importance to test the effectiveness of game-based assessments using objective measures of performance, such as supervisor’s ratings, and a test-retest reliability methodology to establish further the psychometric properties of the new assessment method. Moreover, similar to SJTs, game-based assessments might improve the information gathered about applicants during the selection process as well as applicant reactions (Armstrong et al., Reference Armstrong, Landers, Collmus, Davis and Gangadharbatla2016). Gamification might increase engagement levels which in turn might lead to retention and motivation during the process of selection as well as better predictions about person-job fit (e.g., Chamorro-Premuzic, Akhtar, Winsborough, & Sherman, Reference Chamorro-Premuzic, Akhtar, Winsborough and Sherman2017). Using new technologies and game elements in assessments, recruiters and HR professionals might improve selection decisions making more robust inferences about their performance as game-based assessments do not use self-reported measures that applicants are likely to fake (Fetzer et al., Reference Fetzer, McNamara, Geimer, Goldstein, Pulakos, Passmore and Semedo2017).
Another reason that the use of traditional selection methods might be reconsidered and replaced by new game based tools is that the latter are popular among younger generations. Organizations including game-based assessments into the employee selection process might provide a new technologically advanced experience to applicants sending thus signals about organizational attributes (e.g., innovation) and making the process more fun.
The present study is not without limitations. First of all, performance outcomes were assessed via self-report measures. Although it is suggested that objective measures are the best indicators of individual employee performance, the unavailability of such measurements has forced many previous studies to use self-reported measures of performance (Pransky et al., Reference Pransky, Finkelstein, Berndt, Kyle, Mackell and Tortorice2006). The use of objective measures or supervisor’s report of employee’ performance would lead to more robust findings about the predictive validity of the game-based assessment. Also, some of the GBA’s dimensions were not found to predict performance. One reason might be the use of self-reported measures of performance. “It is likely that self-report and objective measures provide information on distinct, different aspects of work performance. Objective measures, even in jobs that are apparently routine and straightforward, can present challenging levels of complexity, and may provide an estimation of only one dimension of actual job performance.” (Pransky et al., Reference Pransky, Finkelstein, Berndt, Kyle, Mackell and Tortorice2006, p. 396). Future research should explore the ability of the GBA to predict one dimension of performance (e.g., resilience or adaptability) using supervisory ratings or objective performance data.
To establish further the effectiveness of the use of gamification in employee selection, future research should also explore applicants’ reactions. For example, candidates perceive multimedia tests as more valid and enjoyable and as a result, they are more satisfied with the selection process while organizational attractiveness and positive behavioral intentions are increased (Oostrom, Born, & van der Molen, Reference Oostrom, Born, van der Molen, Derks and Bakker2013). The impact of game-based assessments on perceived fairness, organizational attractiveness and job pursuit behaviors should also be investigated to support further their suitability in the selection process. Also, the current study does not address competence and previous experience with technology, which might influence test-takers’ performance. For example, candidates who have experience with on-line games and/or feel competent to use new technology might have less anxiety when new technology is used (Cascio & Montealegre, Reference Cascio and Montealegre2016), and as a result, perform better in a game-based assessment. In general, the limited knowledge and lack of empirical research on the use of gamification in employee selection has made the establishment of a game-based assessment as an effective selection method even more challenging.
Future research should also explore the role of demographic variables on individuals’ performance in game-based assessments. Instead of using demographic variables simply as mere control variables in theory testing, Spector and Brannick (Reference Spector and Brannick2011) suggest to rethink the use of demographics in the first place focusing on the mechanisms that explain relations with demographics rather than on the demographic variables that serve as proxies for the real variables of interest.
Finally, the study might suffer from common method variance effects, since we only used self-reported measures. In order to reduce its effect, we asked the participants to complete the measures in two separate occurrences. Moreover, the Harman’s single factor test we conducted following the guidelines of Podsakoff, Mackenzie, Lee, and Podsakoff (Reference Podsakoff, MacKenzie, Lee and Podsakoff2003) discouraged the impact of common method variance on our results.
Game-based assessments have recently appeared in employee selection calling for further research on their validity. Our study contributes to research on employee selection methods by examining the criterion related validity of a game-based assessment measuring soft skills. Findings of our study indicate that assessments incorporating game elements might predict self-rated job performance, and academic performance, as measured by GPA. Moreover, exploring the incremental validity of the game-based assessment method, we provided evidence that it can predict GPA above and beyond the effect of traditional selection methods, such as personality and cognitive ability tests. These results could change the way organizations and colleges approach traditional assessment methods making the use of gamification in work and academic contexts more widespread in the future.