Skip to main content Accessibility help
×
Home

A new way to look at simulation-based assessment: the relationship between gaze-tracking and exam performance

  • Adam Szulewski (a1), Rylan Egan (a2), Andreas Gegenfurtner (a3), Daniel Howes (a1), Gerhard Dashi (a4), Nick C. J. McGraw (a5), Andrew K. Hall (a1), Damon Dagnone (a1) and Jeroen J. G. van Merrienboer (a6)...

Abstract

Objective

A key task of the team leader in a medical emergency is effective information gathering. Studying information gathering patterns is readily accomplished with the use of gaze-tracking glasses. This technology was used to generate hypotheses about the relationship between performance scores and expert-hypothesized visual areas of interest in residents across scenarios in simulated medical resuscitation examinations.

Methods

Emergency medicine residents wore gaze-tracking glasses during two simulation-based examinations (n=29 and 13 respectively). Blinded experts assessed video-recorded performances using a simulation performance assessment tool that has validity evidence in this context. The relationships between gaze patterns and performance scores were analyzed and potential hypotheses generated. Four scenarios were assessed in this study: diabetic ketoacidosis, bradycardia secondary to beta-blocker overdose, ruptured abdominal aortic aneurysm and metabolic acidosis caused by antifreeze ingestion.

Results

Specific gaze patterns were correlated with objective performance. High performers were more likely to fixate on task-relevant stimuli and appropriately ignore task-irrelevant stimuli compared with lower performers. For example, shorter latency to fixation on the vital signs in a case of diabetic ketoacidosis was positively correlated with performance (r=0.70, p<0.05). Conversely, total time spent fixating on lab values in a case of ruptured abdominal aortic aneurysm was negatively correlated with performance (r= −0.50, p<0.05).

Conclusions

There are differences between the visual patterns of high and low-performing residents. These findings may allow for better characterization of expertise development in resuscitation medicine and provide a framework for future study of visual behaviours in resuscitation cases.

Objectif

Une des tâches importantes du chef d’équipe en médecine d’urgence est la collecte efficace de renseignements, et le port de lunettes de monitorage oculaire permet d’étudier rapidement et facilement les différents modes de collecte visuelle de renseignements. Les chercheurs ont donc eu recours à cette technique pour émettre des hypothèses selon lesquelles il existerait une relation entre la performance des résidents et la saisie de champs visuels d’intérêt présumés tels par des experts, dans des scénarios d’examen de réanimation axés sur la simulation, en médecine.

Méthode

Des résidents en médecine d’urgence ont porté des lunettes de monitorage oculaire au cours de deux séances d’examen par simulation (n=29 et n=13, respectivement). Des experts tenus dans l’ignorance ont évalué le comportement des résidents enregistré sur vidéo, à l’aide d’un outil d’évaluation de la performance axée sur la simulation, dont la validité avait déjà fait ses preuves dans le contexte. Il y a eu analyse de relations entre la collecte visuelle de renseignements et la performance ainsi qu’émission d’hypothèses. Quatre scénarios ont été évalués dans l’étude : l’acidocétose diabétique, une bradycardie consécutive à un surdosage de bêta-bloquants, une rupture d’anévrisme de l’aorte abdominale et une acidose métabolique causée par la consommation d’antigel.

Résultats

Des corrélations ont été établies entre certaines manières de recueillir visuellement les renseignements et la performance objective. Ainsi, ceux qui ont le mieux réussi avaient davantage tendance à se concentrer sur les stimulus pertinents, relatifs aux tâches et à ne pas tenir compte, à juste titre, des stimulus non pertinents comparativement à ceux qui ont moins bien réussi. Par exemple, un temps d’attente raccourci dans la collecte visuelle de renseignements sur les signes vitaux dans le cas de l’acidocétose diabétique s’est révélé en corrélation positive avec la performance (r=0,70; p<0,05), tandis que le temps total écoulé à glaner des renseignements sur les résultats d’examens de laboratoire dans le cas de la rupture d’anévrisme de l’aorte abdominale était en corrélation négative avec la performance (r=−0,50; p<0,05).

Conclusion

Il existe donc des différences entre la manière de glaner visuellement les renseignements et la performance, bonne ou mauvaise, des résidents. Les résultats pourraient permettre une meilleure caractérisation de l’acquisition de la compétence en matière de médecine de réanimation, et fournir un cadre de travail en vue d’études futures sur les comportements visuels dans les cas de réanimation.

Copyright

Corresponding author

Correspondence to: Dr. Adam Szulewski; Assistant Professor, Department of Emergency Medicine, Queen’s University, Kingston General Hospital, Victory 3, 76 Stuart Street, Kingston, ON K7L 2V7; Email: aszulewski@qmed.ca

References

Hide All
1.Kim, J, Neilipovitz, D, Cardinal, P, Chiu, M, Clinch, J. A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: The University of Ottawa Critical Care Medicine, High-Fidelity Simulation, and Crisis Resource Management I Study. Crit Care Med 2006;34(8):2167-2174.10.1097/01.CCM.0000229877.45125.CC
2.Watkins, SC, Roberts, DA, Boulet, JR, McEvoy, MD, Weinger, MB. Evaluation of a Simpler Tool to Assess Nontechnical Skills During Simulated Critical Events. Simul Healthc 2017;12(2):69-75.10.1097/SIH.0000000000000199
3.Szulewski, A, Howes, D. Combining first-person video and gaze-tracking in medical simulation: a technical feasibility study. The Scientific World Journal. 2014;2014.
4.Ericsson, KA, Krampe, RT, Tesch-Römer, C. The role of deliberate practice in the acquisition of expert performance. Psychol Rev 1993;100(3):363.10.1037/0033-295X.100.3.363
5.Kaufman, DR, Arocha, JF, Patel, VL. Expertise and tacit knowledge in medicine. In: Tacit knowledge in professional practice (ed. Sternberg, RJ, Horvath, JA). Psychology Press; 1999: 89-114.
6.Szulewski, A, Brindley, P, Van Merrienboer, JJ. Decision-making during medical crises. In: Brindley P, Cardinal P editors Crisis Resource Management in Acute Care Medicine, 1st ed. Ottawa: Royal College of Physicians and Surgeons Canada; 2017: 36-43.
7.Gaba, DM, Howard, SK, Fish, KJ, Smith, BE, Sowb, YA. Simulation-based training in anesthesia crisis resource management (ACRM): a decade of experience. Simul Gaming 2001;32(2):175-193.10.1177/104687810103200206
8.Issenberg, SB, McGaghie, WC, Petrusa, ER, Lee Gordon, D, Scalese, RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach 2005;27(1):10-28.10.1080/01421590500046924
9.Helle, L, Nivala, M, Kronqvist, P, et al. Traditional microscopy instruction versus process-oriented virtual microscopy instruction: a naturalistic experiment with control group. Diagn Pathol 2011;6(1 Suppl 1):S8.10.1186/1746-1596-6-S1-S8
10.Seppänen, M, Gegenfurtner, A. Seeing through a teacher’s eyes improves students’ imaging interpretation. Med Educ 2012;46(11):1113-1114.10.1111/medu.12041
11.Gegenfurtner, A, Quesada‐Pallarès, C, Knogler, M. Digital simulation‐based training: A meta‐analysis. Br J Educ Technol 2014;45(6):1097-1114.10.1111/bjet.12188
12.Hall, AK, Dagnone, JD, Lacroix, L, Pickett, W, Klinger, DA. Queen’s simulation assessment tool: development and validation of an assessment tool for resuscitation objective structured clinical examination stations in emergency medicine. Simul Healthc 2015;10(2):98-105.10.1097/SIH.0000000000000076
13.Cook, DA, Brydges, R, Zendejas, B, Hamstra, SJ, Hatala, R. Technology-enhanced simulation to assess health professionals: a systematic review of validity evidence, research methods, and reporting quality. Acad Med 2013;88(6):872-883.10.1097/ACM.0b013e31828ffdcf
14.Bick, JS, Demaria, S Jr, Kennedy, JD, et al. Comparison of expert and novice performance of a simulated transesophageal echocardiography examination. Simul Healthc 2013;8(5):329-334.10.1097/SIH.0b013e31829068df
15.Henneman, EA, Marquard, JL, Fisher, DL, Gawlinski, A. Eye tracking: a novel approach for evaluating and improving the safety of healthcare processes in the simulated setting. Simu Healthc 2017;12(1):51-56.
16.Szulewski, A, Roth, N, Howes, D. The use of task-evoked pupillary response as an objective measure of cognitive load in novices and trained physicians: a new tool for the assessment of expertise. Acad Med 2015;90(7):981-987.10.1097/ACM.0000000000000677
17.Gegenfurtner, A, Lehtinen, E, Säljö, R. Expertise differences in the comprehension of visualizations: A meta-analysis of eye-tracking research in professional domains. Educ Psychol Rev 2011;23(4):523-552.10.1007/s10648-011-9174-7
18.Szulewski, A, Gegenfurtner, A, Howes, DW, Sivilotti, ML, van Merriënboer, JJ. Measuring physician cognitive load: validity evidence for a physiologic and a psychometric tool. Adv Health Sci Educ Theory Pract 2017;22(4):951-968.10.1007/s10459-016-9725-2
19.Gegenfurtner, A, Kok, E, van Geel, K, et al. The challenges of studying visual expertise in medical image diagnosis. Med Educ 2017 Jan 51(1):97-104.10.1111/medu.13205
20.Haider, H, Frensch, PA. Eye movement during skill acquisition: more evidence for the information-reduction hypothesis. J Exp Psychol Learn Mem Cogn 1999;25(1):172.10.1037/0278-7393.25.1.172
21.Kok, EM, de Bruin, AB, Robben, SG, van Merriënboer, JJ. Looking in the same manner but seeing it differently: bottom-up and expertise effects in radiology. Appl Cogn Psychol 2012;26(6):854-862.10.1002/acp.2886
22.Haider, H, Frensch, PA. The role of information reduction in skill acquisition. Cogn Psychol 1996;30(3):304-337.10.1006/cogp.1996.0009

Keywords

Type Description Title
WORD
Supplementary materials

Szulewski et al. supplementary material
Appendix S1

 Word (15 KB)
15 KB
WORD
Supplementary materials

Szulewski et al. supplementary material
Appendix S2

 Word (20 KB)
20 KB
WORD
Supplementary materials

Szulewski et al. supplementary material
Appendix S3

 Word (15 KB)
15 KB
WORD
Supplementary materials

Szulewski et al. supplementary material
Appendix S4a-S4c

 Word (27 KB)
27 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed