Skip to main content Accessibility help
×
Home

Development and evaluation of a simulation-based resuscitation scenario assessment tool for emergency medicine residents

Published online by Cambridge University Press:  11 May 2015


Andrew Koch Hall
Affiliation:
Department of Emergency Medicine, Queen's University, Kingston, ON
William Pickett
Affiliation:
Department of Emergency Medicine, Queen's University, Kingston, ON
Jeffrey Damon Dagnone
Affiliation:
Department of Emergency Medicine, Queen's University, Kingston, ON

Abstract

Objective:

We sought to develop and validate a three-station simulation-based Objective Structured Clinical Examination (OSCE) tool to assess emergency medicine resident competency in resuscitation scenarios.

Methods:

An expert panel of emergency physicians developed three scenarios for use with high-fidelity mannequins. For each scenario, a corresponding assessment tool was developed with an essential actions (EA) checklist and a global assessment score (GAS). The scenarios were (1) unstable ventricular tachycardia, (2) respiratory failure, and (3) ST elevation myocardial infarction. Emergency medicine residents were videotaped completing the OSCE, and three clinician experts independently evaluated the videotapes using the assessment tool.

Results:

Twenty-one residents completed the OSCE (nine residents in the College of Family Physicians of Canada– Emergency Medicine [CCFP-EM] program, six junior residents in the Fellow of the Royal College of Physicians of Canada–Emergency Medicine [FRCP-EM] program, six senior residents in the FRCP-EM). Interrater reliability for the EA scores was good but varied between scenarios (Spearman rho 5 [1] 0.68, [2] 0.81, [3] 0.41). Interrater reliability for the GAS was also good, with less variability (rho 5 [1] 0.64, [2] 0.56, [3] 0.62). When comparing GAS scores, senior FRCP residents outperformed CCFP-EM residents in all scenarios and junior residents in two of three scenarios (p , 0.001 to 0.01). Based on EA scores, senior FRCP residents outperformed CCFP-EM residents, but junior residents outperformed senior FRCP residents in scenario 1 and CCFPEM residents in all scenarios (p 5 0.006 to 0.04).

Conclusions:

This study outlines the creation of a high-fidelity simulation assessment tool for trainees in emergency medicine. A single-point GAS demonstrated stronger relational validity and more consistent reliability in comparison with an EA checklist. This preliminary work will provide a foundation for ongoing future development of simulationbased assessment tools.


Type
Original Research • Recherche originale
Copyright
Copyright © Canadian Association of Emergency Physicians 2012

References

1.McLaughlin, S, Fitch, MT, Goyal, DG, et al. Simulation in graduate medical education 2008: a review for emergency medicine. Acad Emerg Med 2008;15:1117–29, doi:10.1111/j.1553-2712.2008.00188.x.CrossRefGoogle ScholarPubMed
2.Miller, GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65(9 Suppl):S63–7.CrossRefGoogle Scholar
3.Epstein, RM. Assessment in medical education. N Engl J Med 2007;356:387–96, doi:10.1056/NEJMe078002.CrossRefGoogle ScholarPubMed
4.Sherbino, J, Bandiera, G, Frank, J. Assessing competence in emergency medicine trainees; an overview of effective methodologies. Can J Emerg Med 2008;10:365.Google ScholarPubMed
5.Lammers, RL, Davenport, M, Korley, F, et al. Teaching and assessing procedural skills using simulation: metrics and methodology. Acad Emerg Med 2008;15:1079–87, doi:10.1111/j.1553-2712.2008.00233.x.CrossRefGoogle ScholarPubMed
6.Boulet, JR. Summative assessment in medicine: the promise of simulation for high-stakes evaluation. Acad Emerg Med 2008;15:1017–24, doi:10.1111/j.1553-2712.2008.00228.x.CrossRefGoogle ScholarPubMed
7.Spillane, L, Hayden, E, Fernandez, R, et al. The assessment of individual cognitive expertise and clinical competency: a research agenda. Acad Emerg Med 2008;15:1071–8,doi:10.1111/j.1553-2712.2008.00271.x.CrossRefGoogle ScholarPubMed
8.Okuda, Y, Bryson, EO, DeMaria, S Jr, et al. The utility of simulation in medical education: what is the evidence? Mt Sinai J Med 2009;76:330–43, doi:10.1002/msj.20127.CrossRefGoogle ScholarPubMed
9.Fritz, PZ, Gray, T, Flanagan, B. Review of mannequin-based high-fidelity simulation in emergency medicine. Emerg Med Australas 2008;20:19, doi:10.1111/j.1742-6723.2007.CrossRefGoogle ScholarPubMed
10.Okuda, Y, Bond, W, Bonfante, G, et al. National growth in simulation training within emergency medicine residency programs, 2003–2008. Acad Emerg Med 2008;15:1113–6, doi:10.1111/j.1553-2712.2008.00195.x.CrossRefGoogle ScholarPubMed
11.Binstadt, ES, Walls, RM, White, BA, et al. A comprehensive medical simulation education curriculum for emergency medicine residents. Ann Emerg Med 2007;49:495, doi:10.1016/j.annemergmed.2006.08.023.CrossRefGoogle ScholarPubMed
12.Philibert, I. Acreditation Council for Graduate Medical Education. ACGME Bulletin. December 2005. Available at: http://www.acgme.org (accessed April 2010).Google Scholar
13.Adler, MD, Trainor, JL, Siddall, VJ, et al. Development and evaluation of high-fidelity simulation case scenarios for pediatric resident education. Ambul Pediatr 2007;7:182–6, doi:10.1016/j.ambp.2006.12.005.CrossRefGoogle ScholarPubMed
14.Brett-Fleegler, MB, Vinci, RJ, Weiner, DL, et al. A simulatorbased tool that assesses pediatric resident resuscitation competency. Pediatrics 2008;121:e597“603, doi:10.1542/peds.2005-1259.CrossRefGoogle ScholarPubMed
15.Morgan, PJ, Cleave-Hogg, DM, Guest, CB, et al. Validity and reliability of undergraduate performance assessments in an anesthesia simulator. Can J Anaesth 2001;48:225–33, doi:10.1007/BF03019750.CrossRefGoogle Scholar
16.Murray, DJ, Boulet, JR, Avidan, M, et al. Performance of residents and anesthesiologists in a simulation-based skill assessment. Anesthesiology 2007;107:705–13, doi:10.1097/01.anes.0000286926.01083.9d.CrossRefGoogle Scholar
17.Weller, JM, Bloch, M, Young, S, et al. Evaluation of high fidelity patient simulator in assessment of performance of anaesthetists. Br J Anaesth 2003;90:43–7, doi:10.1093/bja/90.1.43.CrossRefGoogle ScholarPubMed
18.Berkenstadt, H, Ziv, A, Gafni, N, et al. Incorporating simulation-based objective structured clinical examination into the Israeli National Board Examination in Anesthesiology. Anesth Analg 2006;102:853–8, doi:10.1213/01.ane.0000194934.34552.ab.CrossRefGoogle ScholarPubMed
19.Girzadas, DV Jr, Clay, L, Caris, J, et al. High fidelity simulation can discriminate between novice and experienced residents when assessing competency in patient care. Med Teach 2007;29:452–6, doi:10.1080/01421590701513698.CrossRefGoogle ScholarPubMed
20.Dagnone, JD. Assessment tool for a standardized ventricular fibrillation cardiac arrest. Can J Emerg Med 2009;11:262.Google Scholar
21.Andreatta, PB, Gruppen, LD. Conceptualising and classifying validity evidence for simulation. Med Educ 2009;43:102835, doi:10.1111/j.1365-2923.2009.03454.x.CrossRefGoogle ScholarPubMed
22.Walls, RM. Airway. In: Marks, JA, editor. Rosen’s emergency medicine. Philadelphia: Mosby Elsevier; 2006. p. 226.Google Scholar
23.Morgan, PJ, Cleave-Hogg, D, Guest, CB. A comparison of global ratings and checklist scores from an undergraduate assessment using an anesthesia simulator. Acad Med 2001;76:1053–5, doi:10.1097/00001888-200110000-00016.CrossRefGoogle ScholarPubMed
24.Regehr, G, MacRae, H, Reznick, RK, et al. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 1998;73:993–7, doi:10.1097/00001888-199809000-00020.CrossRefGoogle ScholarPubMed

Full text views

Full text views reflects PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views.

Total number of HTML views: 0
Total number of PDF views: 292 *
View data table for this chart

* Views captured on Cambridge Core between September 2016 - 5th December 2020. This data will be updated every 24 hours.

Access
Hostname: page-component-b4dcdd7-tf8mx Total loading time: 0.369 Render date: 2020-12-05T06:39:28.296Z Query parameters: { "hasAccess": "1", "openAccess": "0", "isLogged": "0", "lang": "en" } Feature Flags last update: Sat Dec 05 2020 05:59:41 GMT+0000 (Coordinated Universal Time) Feature Flags: { "metrics": true, "metricsAbstractViews": false, "peerReview": true, "crossMark": true, "comments": true, "relatedCommentaries": true, "subject": true, "clr": false, "languageSwitch": true }

Send article to Kindle

To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Development and evaluation of a simulation-based resuscitation scenario assessment tool for emergency medicine residents
Available formats
×

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Development and evaluation of a simulation-based resuscitation scenario assessment tool for emergency medicine residents
Available formats
×

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Development and evaluation of a simulation-based resuscitation scenario assessment tool for emergency medicine residents
Available formats
×
×

Reply to: Submit a response


Your details


Conflicting interests

Do you have any conflicting interests? *