Introduction
Disasters or mass-casualty incidents are uncommon events. Responders must be properly trained in order to execute protocols efficiently and treat casualties effectively. This training needs to occur at an individual level for competency of personal skills, an organizational level for capability of agency response, and a systemic level for coordination of overall efforts. According to the World Health Organization (WHO; Geneva, Switzerland) simulation exercise toolbox, there are four types of simulation exercises – tabletop exercise, drill, functional exercise, and full-scale exercise. Table-top exercise is a facilitated discussion of a scenario in a low-stress environment to identify gaps in the response protocol so that improvements can be made. Drill is a supervised activity performed as realistically as possible in order to practice and perfect a particular aspect of the response protocol. Functional exercise is an extensive assessment of the various aspects of an organization’s response protocol in a realistic simulation. 1 Full-scale exercise is the most complex and realistic, as it includes the mobilization of emergency personnel, equipment, and resources across different organizations in order to evaluate the emergency management system in a highly stressful environment which replicates the actual response conditions. 1 Therefore, the most appropriate simulation exercise should be selected based on the objectives of the training, as well as the involvement of individuals, organizations, or systems taking part in it.
Use of simulation in disaster or mass-casualty incident is particularly relevant for two reasons – first, a disaster or a mass-casualty incident is uncommon and simulation is useful as it provides exposure and learning opportunities to uncommon events; and second, a simulated environment resembles actual response environment, allowing learners to be immersed in an experience which facilitates their learning. Reference Al-Elq2–Reference Cannon-Bowers4 When taking part in the simulation, learners can acquire important skills such as interpersonal communication, teamwork, leadership, decision making, prioritization, and stress management – all important when responding to a disaster or mass-casualty incident. Reference Flanagan, Nestel and Joseph5 Learners have also reported perceived self-efficacy and preparedness, as well as increased self-assessed knowledge, confidence, and skills to manage a disaster or mass-casualty incident following simulation exercises. Reference Sweeney, Lui and Watkins6
While simulation is beneficial as an educational tool, the current evidence of its application in full-scale exercises has largely been based on individual reports. Given the immense amount of time, resources, and cost required for the planning, conducting, and after-action-review of full-scale exercises, there is a need to synthesize the available evidence on the effectiveness of simulation so that recommendations can be made for best practices surrounding the use of simulation in full-scale exercises, and suggestions can be made for future works. Ultimately, when training needs are met and learners benefit from the learning experience, the preparedness of the emergency management system should translate to providing better care and achieving improved outcomes for casualties during the response to an actual disaster or mass-casualty incident.
Therefore, the objective of this study is to perform a scoping review on the effectiveness of simulation used in full-scale exercises for response to disasters and mass-casualty incidents world-wide in order to answer the following research question: “How effective is simulation, as assessed in full-scale exercises, for response to disasters and mass-casualty incidents world-wide?” Through this, the authors hope to inform readers how effectiveness has been assessed as reported by individual studies, as well as what are the levels of effectiveness and overall effectiveness as synthesized by this scoping review.
Report
Methods
A scoping review was carried out in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-SCR). 7
Eligibility Criteria—The inclusion criteria were studies on full-scale exercises, as defined in the WHO simulation exercise toolbox, for response to disasters or mass-casualty incidents. 1 Three key components of this definition were used to assess for eligibility, and they were: (1) simulation of a real event and response conditions, (2) mobilization of personnel, equipment, and resources across multiple organizations, as well as (3) evaluation of the plan, coordination, and capability of the emergency management system. The studies had to be published in peer-reviewed journals using the English language from 2001 through 2021. The exclusion criteria were studies on tabletop exercises, drills, and functional exercises, as well as studies that mentioned full-scale exercises but they were not based on the definition in WHO simulation exercise toolbox. Studies involving mass-casualty incidents in military operations were also excluded as the combat nature demands a routine response to mass-casualty incidents in an austere environment which would be different from the non-military context.
Information Sources and Search Strategy—The studies were obtained by searching PubMed (National Center for Biotechnology Information, National Institutes of Health; Bethesda, Maryland USA), Embase (Elsevier; Amsterdam, Netherlands), and Web of Science (Clarivate Analytics; London, United Kingdom) with the help of a university librarian. The search was conducted on July 10, 2022 (Supplementary Material Table 1 for the detailed search strategy; available online only).
Study Selection Process—Two independent reviewers screened for relevant studies using the title and abstract. Short-listed studies were then assessed for eligibility using the full text by the same two independent reviewers. Any conflict was resolved by discussion between the two reviewers to reach a consensus. The reason for excluding studies was recorded.
Data Collection and Analysis—Data were extracted from the studies in a standardized form by one reviewer and cross-checked by a second reviewer before qualitative analysis. The following data were collected:
-
Type of full-scale exercise for response to disaster or mass-casualty incidents (eg, single versus multiple agencies, natural versus man-made disaster, number of casualties, or hazardous material);
-
Type of personnel involved in full-scale exercise (eg, doctors, nurses, paramedics, or students);
-
Type of simulation modality used (eg, low- versus high-fidelity, standardized patients versus mannequin versus task trainers versus hybrid);
-
Objectives (eg, teaching knowledge or skill) of full-scale exercise;
-
Briefing and debriefing with personnel involved in full-scale exercise; and
-
Evaluation for the effectiveness of full-scale exercise (eg, reaction, learning, behavior, results based on Kirkpatrick’s levels of evaluationReference Kirkpatrick8) and reported evaluation outcome.
Results
Among 5,488 articles screened, 20 studies met the inclusion criteria and were included in the scoping review (Figure 1).
Figure 1. PRISMA Flow Diagram.
Study Characteristics—All the included studies were observational, mostly published after 2010 (n = 16; 80%); Figure 2. The studies originated from nine countries with the majority (n = 8; 40%) coming from the United States, followed by Italy (n = 3; 15%). One-quarter of the studies (n = 5; 25%) were published in the journal Prehospital and Disaster Medicine. Characteristics of the included studies are summarized in Table 1. Reference Alexander, Bandiera and Mazurik9–Reference Innis and Mack28
Figure 2. The Trend in the Number of Studies Over the Years.
Table 1. Characteristics of Included Studies
Scenarios of Full-Scale Exercises—Five (25%) involved transportation accidents – aircraft (n = 2), vehicular (n = 2), and train (n = 1). Five (25%) involved hazardous materials – chemical (n = 3), biological (n = 1,) and one with no further details provided. Four (20%) involved natural disasters – earthquakes (n = 2), tornadoes (n = 1), and tsunamis (n = 1). Two (10%) involved terrorists and two (10%) involved active shooters. One (5%) involved a pandemic from Avian flu. While the majority (n = 16; 80%) were single-site exercises, there were four (20%) multi-site exercises, ranging from two to four sites (Supplementary Material Table 2 for details of the scenarios for full-scale exercises; available online only).
Table 2. Agencies and Personnel Involved in Full-Scale Exercises
^ Details of agencies mentioned in part.
* Details of agencies not mentioned at all.
~ Numbers from at least one agency involved.
Agencies and Personnel Involved in Full-Scale Exercises—At least two agencies were involved in the full-scale exercises of the included studies (Table 2). Emergency Medical Services (n = 14; 70%), fire department (n = 13; 65%), hospital (n = 13; 65%), and police department (n = 12; 60%) were most often involved. The personnel involved in the full-scale exercises included participants, observers, controllers, evaluators, simulated casualties, as well as administrative or logistical support staff. However, the exact numbers of personnel involved were often not reported, but when reported, the number ranged from 102 to 990.
Objectives and Briefings of Full-Scale Exercises—The objectives of the full-scale exercises were mentioned in 15 (75%) studies (Table 3) and included one or more of the following domains:
-
Disaster management system standards and gaps;
-
Disaster knowledge, skill, and attitude (eg, triage methods, casualty management);
-
Disaster roles and responsibilities;
-
Disaster crisis resource management, leadership, and teamwork;
-
Disaster collaboration, coordination, and communication;
-
Disaster workflow and protocol (eg, registration, decontamination, or evacuation); and/or
-
Benefits of exercise (eg, gain knowledge or confidence).
Table 3. Objectives and Briefings of Full-Scale Exercises
However, briefings to the participants were only conducted in eight (40%) studies, which covered one or more of the following aspects:
-
General plan of the day;
-
Safety briefing;
-
Education about roles;
-
Familiarization with equipment or team members; and/or
-
Resources available.
In two exercises, participants only had information on the date of the full-scale exercises.
Simulation Modalities Used for Full-Scale Exercises—A simulation was used most commonly in the modality of simulated casualties who were portrayed by students, residents, and volunteers (Table 4). When reported, the number of simulated casualties ranged from 16 to 445. Moulage was used in 12 (60%) exercises. Simulated casualties were briefed on their roles or received cards containing relevant information in eight (40%) exercises. Other simulation modalities include mannequins (n = 3; 15%) and part task trainer (n = 1; 5%).
Table 4. Simulation Modalities Used for Full-Scale Exercises
* Two separate full-scale exercises of 99 and 100 simulated casualties.
^ A total of 18 full-scale exercises over nine days.
Evaluations, Outcomes, and Debriefings of Full-Scale Exercises—Evaluators of the full-scale exercises were often health care professionals or experts with experience in disasters or emergencies, representatives from agencies involved in disasters, as well as faculty members from residency programs or nursing schools (Supplementary Material Table 3 for details; available online only). When reported, the number of evaluators ranged from three to 18. Evaluators were reported to be trained in five (25%) exercises, but only two provided some details on the nature of training. As the evaluation outcomes (Table 5) differed across the included studies, the authors were unable to synthesize them into a single measurement of effectiveness. However, from the report of individual studies, simulation seemed to be generally effective.
Table 5. Evaluations of Full-Scale Exercises
Abbreviations: PICU, pediatric intensive care unit; START, Simple Triage and Rapid Treatment; ED, emergency department; EMS, Emergency Medical Services; MCI, mass-casualty incident; EMT, Emergency Medical Team.
Evaluations of full-scale exercises were performed at more than one Kirkpatrick’s level of evaluation in three (15%) exercises. Evaluation of learning was most common in 15 (75%) exercises and assessed response systems (n = 8), individuals (n = 2), or both (n = 5). Learning of response systems focused on whether guidelines, protocols, and policies were enacted, while learning of individuals focused on knowledge and skill. The evaluations included whether standards, criteria, or indicators were met, assessing the accuracy and providing a score, as well as time taken for tasks to be completed. Two exercises were conducted pre- and post-exercise evaluations at this learning level. Evaluation of participants’ reaction was assessed in seven (35%) exercises, and this involved asking the participants how they felt in domains such as educational value, learning experience, building self-confidence, the realism of exercise, getting them interested, and recommending others to participate. Evaluations of participants’ behaviors and the results on patient outcomes were done only in one (5%) study which examined the performance of tasks and time taken to complete them in an actual mass-casualty incident, and mortality of patients involved, respectively (Table 5).
Debriefings were carried on in five (25%) exercises and covered the following domains (Table 6):
-
Participants’ feelings and experience;
-
Meeting objectives and identifying deficiencies in key areas of roles and responsibilities, crisis resource management, leadership and teamwork, communication, workflow and protocol, patient movement, as well as logistics; and
-
Organization and conduct of full-scale exercises.
Table 6. Debriefings of Full-Scale Exercises
One study mentioned continued feedback for two weeks after on a web-based chat room. However, 11 (55%) of the studies did not mention debriefings and four (20%) did not provide further details about the debriefing process.
Discussion
Full-scale exercises often involve multiple agencies and individuals for the purpose of evaluating the coordination and response capabilities of the emergency management system. In this scoping review, it was found that simulation, with simulated casualties being the most common simulation modality, was generally effective as reported by the included studies. However, this effectiveness was mostly demonstrated at the levels of learning of individuals and system, as well as reaction of individuals based on Kirkpatrick’s levels of evaluation.
Assessing Effectiveness of Simulation in Full-Scale Exercises
Learning of individuals and systems were assessed in full-scale exercises using published standards such as the WHO minimum technical standards for Emergency Medical Teams, or current operating procedures such as the Iranian emergency operation plan. Reference Sheikhbardsiri, Yarmohammadian and Khankeh26,Reference Foo, So and Lu27 Evaluation tools such as the Creighton competence evaluation instrument were also used, but adaptations may be made to better suit the full-scale exercise such as the semi-quantitative performance indicators from the Center for Teaching and Research in Disaster Medicine and Traumatology (University of Linköping, Linköping, Sweden) which were modified by Center for Research and Training in Disaster Medicine, Humanitarian Aid, and Global Health (CRIMEDIM; Università del Piemonte Orientale, Novara, Italy). Reference Djalali, Carenzo and Ragazzoni19 Other measures of learning include the accuracy of tasks performed, such as the proportion of patients who were correctly triaged or the appropriateness of treatment rendered, as well as the time is taken to do so such as time for triage, treatment, or evacuation to hospital.
Reaction of individuals was assessed in full-scale exercises using completion of questionnaires, surveys, or feedback forms, as well as participation in interviews or focus group discussions. The corresponding perceptions of participants in various domains were reported mostly quantitatively rather than qualitatively. Among the domains assessed, simulation in full-scale exercises most commonly demonstrated a perception of high educational value in instilling knowledge, skill, and attitude both at an individual level and as an interprofessional team. This was followed by perceptions of simulation in full-scale exercises being an enjoyable learning experience which participants were satisfied with, and how self-confidence as a responder to disasters and mass-casualty incidents was improved after the full-scale exercises.
Evaluating the effectiveness of simulation in full-scale exercises at the higher Kirkpatrick levels of behavior and results would be most valuable as assessments could be carried out when participants perform their roles as responders in effective and efficient emergency management systems. Their behavior during an actual disaster or mass-casualty incident would be indicative of their readiness to translate knowledge, skill, and attitude into clinical practice when attending to casualties. The results of their actions would eventually impact outcomes of casualties ranging from immediate ones, such as mortality, to short-term ones, such as morbidity or survival to hospital discharge, and long-term ones, such as return to previous function and general well-being. However, this would be challenging to evaluate given the infrequent occurrences, and the challenge in standardizing outcomes to measure due to the varied nature of disasters and mass-casualty incidents. There was only a single report by Lenz, et al on the behavior of the responders and outcomes of the casualties during an active threat resolution at the STEM School shooting in Highlands Ranch, Colorado USA, which occurred six months after the full-scale exercise. Reference Lenz, Shelton and Ryznar24
Improving the Effectiveness of Simulation in Full-Scale Exercises
Full-scale exercises are manpower and resource intensive. They are costly in terms of finances and time required from planning to execution. In order to improve the effectiveness of simulation in full-scale exercises, the key consideration is that full-scale exercises should be integrated into a training program consisting of simulation exercises that are progressively complex – starting from tabletop exercises, moving on to drills, functional exercises, and eventually full-scale exercises. This stepwise approach can facilitate basic learning for individuals and single organizations at the earlier stages of simulation exercises and advance learning for multiple organizations and national systems at the later stages of simulation exercises. Full-scale exercise should ideally not be carried out as a stand-alone exercise without prior simulation exercises to build upon. However, in this scoping review, the authors were unable to ascertain if there were prior table-top exercises, drills, or functional exercises before these full-scale exercises were conducted.
The effectiveness of simulation in full-scale exercises could also be enhanced further by adhering to principles of learning theories and applying evidence-based practices for the conduct of simulation. Reference Wang29,Reference Sittner, Aebersold and Paige30 Full-scale exercises could be anchored on adult learning principles which emphasize that the learning objectives should be relevant and tailored to work experience so that participants can learn in a safe environment provided by simulation. Learning needs should be assessed in order to define learning objectives and create scenarios for full-scale exercises. The simulation modality and the level of fidelity should be determined. Participants should undergo briefing and be given time for familiarization with the simulation modality and setting. Simulated casualties should be trained, and evaluators should be standardized in their assessment. A participant-centered approach of providing feedback and debriefing should be adopted to facilitate reflective learning. In this scoping review, these best practices of the simulation were inconsistently reported across the included studies. However, the authors were unable to determine if these best practices were excluded in the conducting of full-scale exercises or in the reporting process. Therefore, a framework could be used for the conducting, which may improve effectiveness of simulation during full-scale exercises, and the same framework can be used for reporting, which may enhance documenting the effectiveness of simulation after full-scale exercises.
Increased Use of Simulation in Full-Scale Exercises
The use of simulation in full-scale exercises rose after 2010, and this could be attributed to two key drivers. Firstly, the increased frequency of disasters and mass-casualty incidents with higher morbidity and mortality rates has called to attention the need for training organizations and personnel through the conduct of full-scale exercises during the preparedness phase of disaster management. Reference Hodgson31 The adequacy of response could also be assessed so that gaps could be identified and actions could then be instituted at national, organizational, and individual levels for improvement. The second driver is the tremendous growth of simulation as a training modality. Anchored in learning theories and guided by evidence-based practices, simulation has wide applications in health care settings and is particularly valuable for training related to disaster and mass-casualty incidents as simulation could provide realism in a safe environment for the participants. Reference Herrera-Aliaga and Estrada32
Yet, from this scoping review, despite the surge in the literature on full-scale exercises, reports on outcomes of effectiveness were limited, as well as variable in terms of what constituted effectiveness and how the effectiveness was measured. The authors were unable to determine any evolving trends across the years in participants involved, objectives identified, simulation modalities utilized, evaluation methods employed, outcomes assessed, as well as briefings and debriefings conducted, as these were inconsistently reported in the included studies.
Recommendations for Use of Simulation in Full-Scale Exercises
Simulation in full-scale exercises should be well-conducted before its effectiveness could be accurately measured. Likewise, reports of simulation in full-scale exercises should be of good quality. Reference Cheng, Kessler and Mackinnon33 Future works on the use of simulation in full-scale exercises should first include standardized reporting of methodologies used in the conduct of simulation. This should be done using a framework which would allow for assessment of the quality of simulation based on six key domains which reflect best evidence-based practices – participants involved, objectives identified, simulation modalities utilized, evaluation methods employed, outcomes assessed, as well as briefings and debriefings conducted. Using this framework, key aspects of simulation use for full-scale exercises could be consistently reported to develop and advance knowledge in this field – important details may be less likely to be left out, and this may allow for comparisons between individual studies. Novel or modified methodologies could be highlighted so that evidence-based practices specific to the context of full-scale exercise could be augmented within the community.
Focusing on effectiveness of simulation in full-scale exercises, future works should clearly identify Kirkpatrick’s levels of evaluation on effectiveness, and explicitly define how evaluation would be carried out for the particular level, such as self-reporting using survey or feedback through focus group discussion for reaction of individuals, benchmarking against published standards or specific protocols for learning of system or individuals, responding to casualties competently and confidently for behavior of system or individuals, as well as reporting patient-centered outcomes of casualties for results. It would also be important to provide information about how these evaluation tools are selected, how evaluation outcomes for measuring effectiveness are selected, whether these tools and outcomes have been previously used, and whether they have been validated for accuracy and precision in evaluating the effectiveness of simulation in full-scale exercises.
Limitations
Firstly, there was no universal gold standard definition for full-scale exercise, so the definition in WHO simulation exercise toolbox was used in this scoping review. In the study selection process, there were articles which were reported as full-scale exercises but did not demonstrate all three key components of the definition in WHO simulation exercise toolbox. This observation called for a need for a universal gold standard definition for full-scale exercise to be agreed upon in the community and then be used consistently for reporting in the literature.
Next, non-English articles were excluded, and this may have led to useful data being missed by this scoping review. Nonetheless, the authors were able to understand the body of evidence available to address the research question, and share the findings, gaps, and recommendations so that subsequent works in this field can benefit from this. Also, many methods exist to evaluate the effectiveness of training. In addition to Kirkpatrick’s levels of evaluation, other methods such as the Context, Input, Process, and Product model and Kaufman’s model of learning evaluation exist, all with their pros and cons. Reference Toosi, Modarres, Amini and Geranmayeh34,Reference Kaufman and Keller35 While Kirkpatrick’s levels may be overtly focused on outcomes instead processes of learning, and the linear causality implied across the levels may be too simplistic, it is still widely recognized and accepted in simulation and health care education, thus making it a good standard to base this scoping review on. Reference Johnston, Coyer and Nash36 Full-scale exercises may involve learning of both individuals and response systems, therefore, the authors had to modify Kirkpatrick’s level of learning and delineate whether individuals or response systems were assessed in the evaluation.
Finally, while the simulation was shown to be overall effective, the respective domains which reflect the best evidence-based practice, such as briefing and debriefing of participants, or choice of simulation modality and fidelity, could not be assessed individually to determine which aspects of the conduct of simulation contributed most to its effectiveness in full-scale exercises.
Conclusion
Full-scale exercises provide training opportunities for disasters and mass-casualty incidents, which are rare events. In the simulated setting, which is created to replicate the actual environment, participants can attend to simulated casualties in accordance with existing protocols. However, these training are labor and resource intensive, and they often require significant amounts of time for planning, coordination, and execution. This scoping review has shown that the use of simulation in full-scale exercises, based on reports by individual studies, is generally effective but has mostly been reported at lower Kirkpatrick’s levels of reaction of individuals, as well as learning of individuals and/or systems. Best evidence-based practices for simulation should be adhered to in full-scale exercises so that the training would adequately prepare the participants and eventually translate into better care and outcomes for casualties during an actual disaster or mass-casualty incident. In addition, the reporting of simulation use in full-scale exercises should be standardized using a framework, and the evaluation process should be rigorous so that effectiveness could be determined and compared across full-scale exercises.
Conflicts of interest/funding
All authors do not have any financial disclosures or conflicts of interest.
Supplementary Materials
To view supplementary material for this article, please visit https://doi.org/10.1017/S1049023X2300660X
