Hostname: page-component-7c8c6479df-xxrs7 Total loading time: 0 Render date: 2024-03-19T02:54:27.168Z Has data issue: false hasContentIssue false

Development of a game-based learning tool for applied team science communication in a virtual clinical trial

Published online by Cambridge University Press:  13 September 2018

Colin A. Depp*
Affiliation:
Altman Clinical and Translational Research Institute (ACTRI), University of California San Diego, San Diego, CA, USA
Alex Howland
Affiliation:
VirBELA LLC, La Jolla, CA, USA
Jill Dumbauld
Affiliation:
Altman Clinical and Translational Research Institute (ACTRI), University of California San Diego, San Diego, CA, USA
John Fontanesi
Affiliation:
Altman Clinical and Translational Research Institute (ACTRI), University of California San Diego, San Diego, CA, USA
David Firestein
Affiliation:
Altman Clinical and Translational Research Institute (ACTRI), University of California San Diego, San Diego, CA, USA
Gary S. Firestein
Affiliation:
Altman Clinical and Translational Research Institute (ACTRI), University of California San Diego, San Diego, CA, USA
*
*Address for correspondence: C. A. Depp, PhD, Altman Clinical and Translational Research Institute, University of California San Diego, 9500 Gilman Drive, M/C 0664 La Jolla, San Diego, CA 92093, USA. (Email: cdepp@ucsd.edu)
Rights & Permissions [Opens in a new window]

Abstract

Educational tools for application of team science competencies in clinical research are needed. Our interdisciplinary group developed and evaluated acceptability of a virtual world game-based learning tool simulating a multisite clinical trial; performance hinges on effective intrateam communication. Initial implementation with clinical research trainees (n=40) indicates high satisfaction and perceived relevance to team science and research career goals. Game-based learning may play an important role in team science training.

Type
Education
Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NCCreative Common License - ND
This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is unaltered and is properly cited. The written permission of Cambridge University Press must be obtained for commercial re-use or in order to create a derivative work
Copyright
© The Association for Clinical and Translational Science 2018

Introduction

Team science in clinical and translational research is increasingly required for complex multidisciplinary projects [1]. Because this discipline is a relatively new field, the competencies for team science learners are not well defined [Reference Begg2]. Moreover, a recent analysis of Clinical and Translational Science Award institution curricula indicated that available didactic resources for team science training were adapted from other disciplines and not directly pertinent to clinical research [Reference Calvin-Naylor3]. In addition, best practices for training in the application of team science competencies are lacking, such as facilitated practice in interprofessional communication. Therefore, there is a need for educational tools that foster skills for effective clinical research teams.

Game-based learning, particularly through digital media, can increase motivation to learn and to build skills in several fields [Reference Clark, Tanner-Smith and Killingsworth4, Reference Qian and Clark5]. A number of recent reviews indicate positive impact of games in knowledge acquisition and skill development as compared with traditional teaching methods [Reference Clark, Tanner-Smith and Killingsworth4, Reference Qian and Clark5]. Skills in interprofessional communication are fundamental to team science competencies, and games can potentially connect group communication to immediate performance feedback. Team games also allow participants to assess the relative value of strategies for collaboration, including decisions regarding assignment of leadership, attunement to individual- Versus team-level performance metrics, and strategies for collaborative planning.

Virtual worlds afford opportunities to simulate applied scenarios, and, in distance learning, can enhance the social element of didactic content. Game-based learning in virtual worlds to support teamwork has been used in management [Reference Ellis6, Reference Li, D’Souza and Yunfei7], and medical teaching [Reference Dev8]. To explore the potential of this methodology in clinical research, we designed, developed and evaluated acceptability of a team science education game centered on communication and collaboration in a multisite clinical trial. We also describe key decision points during development, which might help educators develop other clinical and translational research education games.

Methods and Results

Design Phase

We established an interdisciplinary design group that included clinical research educators, computer scientists, organizational psychologists, instructional designers, graphic artists, and end users. The group engaged in an 8-month iterative development process, building from a partnership between UC San Diego Altman Clinical and Translational Research Institute and VirBELA, Inc., a creator of virtual worlds for collaboration and education. The design group’s initial plan was to develop a virtual world that addressed broad team science and leadership competencies, with a series of “mini-games” that simulated multiple steps in the clinical trial process (e.g., staff selection, Gantt charts). These ideas were presented to our targeted end users (i.e., predoctoral and postdoctoral scholars and junior faculty engaged in clinical research training and their instructors) with storyboards, and feedback revealed tensions between devising a simulation experience to teach the elements of clinical trials Versus gameplay that focused more on the interactions and strategies for communication. Feedback indicated that linking communication and collaboration strategies with performance in a stylized clinical trial would best capitalize on the unique strengths of the virtual world. Moreover, the initial design was intended to connect progressive experiences over multiple sessions. Pragmatic considerations regarding feasibility led to the conclusion that the most scalable approach would be a shorter game that could be embedded in multiple courses containing team science content. We then developed a facilitator guide and didactic content connecting game experience with the learning objectives.

The choice of a multisite clinical trial as a setting was selected over other options (e.g., collaborative planning for a novel study) because of the variety of roles in a clinical trial, applicability to broad range of learners, and the inherent necessity to simultaneously consider individual, “site” and trial-wide performance in a multisite trial. Detailed game elements, including the duration, perspective, scoring and performance metrics were specified in a game concept document [Reference Ibrahim and Jaafar9]. Assessment and refinement of the beta-version involved about 15 meetings with 40 end users and administrators, and over 500 hours of software development time.

Game Learning Objectives and Design

The learning objectives of the completed game are to (1) identify interdependencies between professionals in a clinical trial needed to achieve a shared goal, (2) reflect on communication as a critical component of individual role performance to support team-level performance, and (3) learn strategies for using trial accrual data to facilitate team coordination discussions aimed for performance improvement.

The game is played in a virtual world 3D campus programmed with the commercially available Unity 3D engine. Learners first download the application (available for Macintosh or Windows operating systems). They next register an account and create an avatar to represent themselves. After team members view game-play instructions, they enter the game, which can be calibrated to last either 5 or 10 minutes. The object of the game is to recruit, enroll, and complete the protocol for study participants during a timed period. The game is played by teams of 4 learners, with each individual team member taking 1 of the 4 “roles” which represent professionals in a clinical trial: Recruiters, who are responsible for identifying and prescreening potential study participants among a general population; Screeners, who identify eligibility of recruited individuals based on inclusion and exclusion criteria; Coordinators, who manage the day-to-day completion of study-related activities; and Principal Investigators (PIs), who use budget-limited resources to improve processes and manage overall team performance (see Fig. 1). Each team may also include an optional nonparticipant Observer who does not play a role but is present in the virtual world. Each player works semi-independently in a distinct virtual space (with the exception of the PI, who is able to navigate freely throughout the virtual setting). The minimum number of players is 4, and up to 4 teams or “sites” may play the game for a maximum of 16 simultaneous players.

Fig. 1 Screenshots of team roles. From top left: Recruiter, Screener, Coordinator, Principal Investigator. At the top of each screen, time remaining, budget, and enrollment metrics are displayed.

Although players are tied to their individual tasks, they are able and encouraged to communicate with their teammates using Voice over IP. The game is designed with several potential opportunities for “bottlenecks” in managing study flow, as well as several relevant points of information that are not revealed fully to all players, such as the impact of various recruitment strategies on the complexity of the screener’s tasks and the negative impact of participant “wait times” on attrition. Thus, intrateam communication is critical in successfully completing the game. Once the allotted time has run out, team members reconvene in a debrief “room,” in which they view indicators of their performance in the clinical trial, including the number of participants recruited, screened, retained and completed (see Fig. 2). Learners can compare their outcome with other teams, introducing competition to increase engagement. The teams can play multiple times and experiment with different communication and teamwork strategies in order to improve their performance.

Fig. 2 Screenshot of the debrief room. After completion of the game, team performance metrics are displayed and discussed in comparison with other teams.

In-Class Facilitation

We developed the game to be played within a 1.5–2-hour facilitated discussion, with discussion connecting game experience with relevant principles in team science. The following in-class agenda was employed:

  1. (1) Group discussion (10 min): learners first reflect on their experiences in effective and ineffective teams, in research teams or, without such experience, in clinical teams.

  2. (2) Team science and clinical trials (10 min): trends in increased prevalence of multiauthored manuscripts and multi-PI funding proposals [1], as well as the increasing complexity of research projects, are discussed. Typologies of scientific teams, from unidisciplinary to interdisciplinary and transdisciplinary [1] provide context for trainees to reflect on their current clinical and research teams.

  3. (3) Communication and coordination in team science (10 min): using a brief case study in workplace communication [Reference Pentland10], facilitators and trainees discuss the effect of communication in team performance. Common conflicts and challenges in team science [Reference Zucker11], as well as potential mitigation strategies, are discussed.

  4. (4) Gameplay (30–40 min): two 5 or 10 minute rounds of the game are played, with time for debrief and strategy-building between rounds. Mid-point debrief discussion elucidates pitfalls in gameplay and communication, strategies for improved performance, and team dynamics.

  5. (5) Final debrief (10 min): the facilitator guides trainees through a discussion of the generalization of their game experience with “real-world” teams.

  6. (6) Team science resources (10 min): the discussion ends with additional reading and topical resources available for trainees, which include an NIH-sponsored toolkits and field guides [Reference Bennett, Gadlin and Levine-Finley12, 13], resources to assess team readiness and performance [Reference Goldsmith and Morgan14, Reference Olson and Olson15] and conflict mitigation information [Reference Gadlin and Jessar16].

The freely downloadable software program, facilitator guide, and didactic materials are available through http://www.ctri.ucsd.edu/education/Pages/GameBasedLearning.aspx.

Evaluation, Analysis, and Results

Our initial focus of evaluation is acceptability, and we have implemented the game in different groups of learners as a single session embedded in research training curricula at UC San Diego, including: (1) a career development seminar for MD or PhD junior faculty in clinical research, such as KL2 scholars (n=15), (2) a credit-bearing applied lab-based course in a Master’s in Clinical Research with medical students (n=20), and (3) an interdisciplinary T32 training program for junior faculty members with MDs or PhDs (n=5). Evaluations were gathered voluntarily and anonymously via Qualtrics, eliciting Likert-type satisfaction ratings and open-ended questions soliciting perceived benefits and ideas for improvement and future application; our primary outcome was user satisfaction with the class session. Given the focus on acceptability, analyses were descriptive and no data on current or prior team participation or long-term outcomes were collected. These data were collected anonymously as part of routine evaluations and therefore the study was not subject to human research protections. Data analysis was approved by the UC San Diego HRPP, under Exempt Status.

Responses indicate a high level of satisfaction with the session (mean of 8.8 out 10, SD=1.6), perceived relevance to the understanding the topic of team science, as well as usefulness to research careers (Table 1). Qualitative feedback aligned with the learning objectives in providing an applied experience in coordination and communications. Suggestions for improvement included modifications to game instructions and aspects of the user controls. Learners identified potential extensions of the game in the context of stimulating discussion on roles and communication in actual research teams.

Table 1 Scholar evaluation data (n=19)

Discussion

Clinical and translational research is increasingly a “team sport,” and our online game-based learning tool delivers an applied learning experience tailored to clinical research learners. Extensive discussions with learners, simulation and game experts, and education experts led us to modify our original plan for a simulation in favor of a “gamified” solution. As a result of these modifications, the primary goal of the tool was refocused to collaboration and teamwork. Our game-based tool contains relevant content in clinical trial implementation, but the learning objectives are focused on communication skills and strategies fundamental to team science. Simple game elements such as scores and competition enhance the experience and increase motivation to improve performance.

Our results support feasibility, acceptability, and perceived relevance to team science in small sample of learners. While this is encouraging, quantitative and qualitative data on the impact of the game on skill acquisition and team behavior are needed, as an approach to problem-based learning in clinical and translational research education [Reference Calvin-Naylor3]. The field is still in its infancy, and there is a need to define which learning objectives are best suited for games, establish best practices for evaluating impact on competencies, and optimize methods for enhancing transfer of training to real-world experience. Our implementation of game-based learning integrated live instructor facilitation, and therefore future research could evaluate whether automated approaches to facilitation (such as written reflection) could be employed to increase scalability. The application we described was targeted to early-stage learners in clinical research, and we are identifying how games might foster skills in existing teams, as well as among clinical research coordinators whose learning needs are poorly understood [Reference Calvin-Naylor3]. We are also identifying data that can be extracted from game-based learning sessions, since important aspects of communication can be objectively measured (e.g., team communication patterns). These objective data could form the basis for performance-based assessment and real-time feedback [Reference Howland17]. Finally, our development process highlighted the importance of an interdisciplinary approach to program development, use of game concept documentation, and an iterative development process with frequent end-user engagement.

Acknowledgments

The authors thank Wes Hawkins and Erik Hill who were the lead programmers on this project, along with the scholars and faculty members who provided invaluable feedback.

Financial Support

The project described was partially supported by the National Institutes of Health, Grants UL1TR001442 and KL2TR001444.

Disclosures

A.H. is CEO of VirBELA LLC, which programmed software through a contract with UCSD ACTRI. Though VirBELA LLC has a commercial interest in its other virtual worlds applications, the game is available for distribution free of charge. All other authors have no conflicts of interest to declare.

References

1. National Research Council. Enhancing the Effectiveness of Team Science. Washington, DC: The National Academies Press, 2015.Google Scholar
2. Begg, MD, et al. Approaches to preparing young scholars for careers in interdisciplinary team science. Journal of Investigative Medicine 2014; 62: 1425.Google Scholar
3. Calvin-Naylor, NA, et al. Education and training of clinical and translational study investigators and research coordinators: a competency-based approach. Journal of Clinical and Translational Science 2017; 1: 1625.Google Scholar
4. Clark, DB, Tanner-Smith, EE, Killingsworth, SS. Digital games, design, and learning: a systematic review and meta-analysis. Review of Educational Research 2016; 86: 79122.Google Scholar
5. Qian, M, Clark, KR. Game-based learning and 21st century skills: a review of recent research. Computers in Human Behavior 2016; 63: 5058.Google Scholar
6. Ellis, JB, et al.Games for virtual team building. In: Proceedings of the 7th ACM Conference on Designing Interactive Systems. Cape Town, South Africa: ACM, 2008, pp. 295–304.Google Scholar
7. Li, J, D’Souza, D, Yunfei, D. Exploring the contribution of virtual worlds to learning in organizations. Human Resource Development Review 2011; 10: 264285.Google Scholar
8. Dev, P, et al. Virtual worlds and team training. Anesthesiology Clinics 2007; 25: 321336.Google Scholar
9. Ibrahim, R, Jaafar, A. Educational games (EG) design framework: Combination of game design, pedagogy, and content modeling. In: Proceedings of the 2009 International Conference on Electrical Engineering and Informatics, ICEEI 2009. Washington, DC: IEEE, 2009, pp. 293298.Google Scholar
10. Pentland, AS. The new science of building great teams. Harvard Business Review, 2012.Google Scholar
11. Zucker, D. Developing your career in an age of team science. Journal of Investigative Medicine 2012; 60: 779784.Google Scholar
12. Bennett, LM, Gadlin, H, Levine-Finley, S. Collaboration and Team Science: A Field Guide. Washington, DC: National Institutes of Health, 2010.Google Scholar
13. National Cancer Institute. Team Science Toolkit. Washington, DC: National Institutes of Health.Google Scholar
14. Goldsmith, M, Morgan, H. Team building without time wasting. In: Goldsmith M, Lyons L, Freas A, eds. Coaching for Leadership: How the World's Greatest Coaches Help Leaders Learn. San Francisco, CA: Jossey-Bass, 2000, pp. 103109.Google Scholar
15. Olson, G, Olson, J. Collaboration Success Wizard [Internet], 2017 [cited Oct 20, 2017]. (http://hana.ics.uci.edu/wizard/)Google Scholar
16. Gadlin, H, Jessar, K. Preempting discord: prenuptial agreements for scientists. The NIH Catalyst 2002; 10: 13.Google Scholar
17. Howland, AC, et al. Developing a virtual assessment center. Consulting Psychology Journal: Practice and Research 2015; 67: 110126.Google Scholar
Figure 0

Fig. 1 Screenshots of team roles. From top left: Recruiter, Screener, Coordinator, Principal Investigator. At the top of each screen, time remaining, budget, and enrollment metrics are displayed.

Figure 1

Fig. 2 Screenshot of the debrief room. After completion of the game, team performance metrics are displayed and discussed in comparison with other teams.

Figure 2

Table 1 Scholar evaluation data (n=19)