Skip to main content Accessibility help
×
Home

Information:

  • Access
  • Cited by 4

Actions:

      • Send article to Kindle

        To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Effect of a point-of-care ultrasound protocol on the diagnostic performance of medical learners during simulated cardiorespiratory scenarios
        Available formats
        ×

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Effect of a point-of-care ultrasound protocol on the diagnostic performance of medical learners during simulated cardiorespiratory scenarios
        Available formats
        ×

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Effect of a point-of-care ultrasound protocol on the diagnostic performance of medical learners during simulated cardiorespiratory scenarios
        Available formats
        ×
Export citation

Abstract

Background

Goal-directed point-of-care ultrasound (PoCUS) protocols have been shown to improve the diagnostic accuracy of the initial clinical assessment of the critically ill patient. The diagnostic impact of the Abdominal and Cardiac Evaluation with Sonography in Shock (ACES) protocol was assessed in simulated emergency medical scenarios.

Methods

Following a focused PoCUS training program, the diagnostic accuracy, confidence, and precision of 12 medical learners participating in standardized scenarios were tested using high-fidelity clinical and ultrasound simulators. Participants were assessed during 72 simulated cardiorespiratory scenarios. Differential diagnoses were collected from participants before and after PoCUS in each scenario, and confidence surveys were completed. Data were analysed using R software.

Results

Prior to PoCUS, 45 (62.5%) correct primary diagnoses were made compared with 64 (88.9%) following PoCUS (χ2=14, 1df, p=0.0002). PoCUS was also shown to increase participants’ confidence in their diagnoses. The mean confidence in diagnosis score pre-PoCUS was 52.2 (SD=14.7), whereas post-PoCUS it was 81.7 (SD=9.5). The estimated difference in means (−28.36) was significant (t=−7.71, p<0.0001). Using PoCUS, participants were further able to narrow their differential diagnoses. The median number of diagnoses for each patient pre-PoCUS was 3.5 (interquartile range [IQR]=3.8, 3.0) with a median of 2.3 (IQR=2.9,1.5) diagnoses post-PoCUS. The difference was significant (W=0, p<0.001).

Conclusion

This pilot study suggests that, in medical learners newly competent in PoCUS, the addition of an ACES PoCUS protocol to standard clinical assessment improves diagnostic accuracy, confidence, and precision in simulated cardiorespiratory scenarios. This is consistent with clinical studies and supports the use of ultrasound during medical simulation.

Background

Point-of-care ultrasound (PoCUS) and the use of simulation in training have become an integral part of emergency medicine practice. PoCUS is a core competency in emergency medicine and other clinical specialties, and gives physicians the ability to rapidly and noninvasively assess patient physiology, anatomy and pathology, thus improving diagnosis and guiding resuscitation. 1 , 2 Emergency PoCUS has been shown to help differentiate causes of pleuritic chest pain in patients with a normal chest x-ray, 3 to improve diagnostic accuracy and narrow the differential diagnosis in patients with undifferentiated hypotension, 4 to improve time-to-diagnosis and mortality of cardiac trauma patients, 5 and to offer benefit in the accurate and timely diagnosis of acute cardiorespiratory emergencies, 6 , 7 in addition to other documented benefits. 8

Numerous studies have shown that competence in such PoCUS protocols, including the Focused Assessment with Sonography for Trauma 9 and Prehospital Assessment with Ultrasound for Emergencies 10 can be achieved after short intensive training. The Abdominal and Cardiac Evaluation with Sonography in Shock (ACES) protocol, previously described by Atkinson et al. in 2009, 11 is a structured six-view PoCUS approach for patients in shock (Figure 1). It was recently found that medical learners demonstrated competency in simulated PoCUS for the ACES protocol after a focused training program involving specified reading, videos, and an interactive workshop. Competency was measured by image acquisition and interpretation, with learners successfully generating 99.7% of the ultrasound windows and correctly interpreting 93% of the scans. 12

Figure 1 The Abdominal and Cardiothoracic Evaluation with Sonography in Shock (ACES) protocol: ultrasound windows consist of 1) one or more cardiac views, 2) an inferior vena cava view, 3) a screen of the abdominal aorta, 4) right and 5) left flank views for pleural and peritoneal fluid, and 6) a pelvic view for bladder size and free fluid.

High-fidelity simulation offers many benefits in medical education, including decreased reliance on patient or volunteer availability, and the ability to customize and repeat virtually any clinical scenario. Unfortunately, the fidelity, or realism, of the simulated human body is restricted by the available technologies, which limits the ability to realistically approximate actual ultrasound images. 13

The purpose of this study was to investigate the impact of integrating simulated PoCUS into standard human patient simulation by combining high-fidelity clinical and PoCUS simulators. The primary objective was to determine whether medical learners who have completed training in simulated PoCUS demonstrate an improvement in their diagnostic accuracy, confidence, and precision during simulated cardiorespiratory emergency scenarios with the addition of ultrasound into simulation (USIM).

Methods

Study design

This was a prospective observational study involving medical school clerks (third- and fourth-year medical students) and resident physicians (PGY1–3) completing rotations in an urban regional hospital. The study was approved by the Dalhousie University Research Ethics Board.

Study setting and population

The study was conducted in the simulation centre located in a multidisciplinary teaching facility at the New Brunswick Community College from March to September 2012. All participants enrolled voluntarily, after recruitment using local communication channels and signed consent and confidentiality forms. Inclusion criteria were 1) voluntary participation, 2) being available and willing to attend the training and testing session, and 3) being either a third- or fourth-year medical student clerk or a physician currently in residency training. Participants had no prior ultrasound training. Because this was a pilot study, no power calculation was undertaken.

Study protocol

All participants underwent a standardized training protocol in simulated PoCUS, involving pre-reading, didactic teaching, and supervised hands-on experience with a high-fidelity CAE Vimedix ultrasound simulator (Figure 2). The training protocol we utilized has been described previously by our group. 12

Figure 2 Picture of the CAE Vimedix high-fidelity ultrasound simulator used in the study.

Following PoCUS training, participants were introduced to the patient simulation suite and the high-fidelity clinical simulator, the Laerdal SimMan 3G. During this time, participants were oriented to the simulator and interviewed the simulated patient through the built-in microphone, listened to heart and lung sounds, familiarized themselves with the monitor, and identified decreased pulses and diaphoresis. Each of the participants then participated in a full orientation scenario to solidify their confidence with the simulators. Training continued until the participants felt comfortable with the simulation setting and agreed to proceed to the testing scenarios.

To determine whether the addition of PoCUS to standardized simulated scenarios improves diagnosis, each participant completed a set of six cardiorespiratory medical scenarios. Each of these scenarios included pathology that was potentially detectable by PoCUS, including aortic stenosis, cardiac tamponade, dilated cardiomyopathy, myocardial infarction, pleural effusion, and pulmonary embolism (features of the simulators for each scenario are included in Appendix 1). Each of the 12 participants completed all six scenarios for a total of 72 datasets. None of the participants had advance knowledge of the diagnosis in the case that they were managing. The sequence of scenario delivery was generated using a Latin square design to ensure a counterbalanced approach. This arrangement was intended to minimize carryover effects due to practice and learning. Participants were instructed to consider only the diagnosis and were not expected to provide any interventions or treatment options for the simulated patient.

For each scenario, the simulation suite was equipped with both simulators (programmed for the given scenario), a confederate role-playing nurse who presented the case to the participant, and a confederate in an adjacent room playing the role of the patient via a built-in microphone in the high-fidelity clinical simulator. Scenarios began with the participant being provided with a clinical summary by the nurse. This summary included the patient’s presenting complaint, vitals, and an electrocardiogram. The participant would then approach the high-fidelity clinical simulator, perform a focused history and physical exam, and then formulate a differential diagnosis. Participants were allowed a maximum of five possible diagnoses, ranked in order of decreasing confidence of likelihood and given a numeric percentage between 1% and 100%.

Following clinical examination, the participant would proceed to the PoCUS simulator and perform an ACES scan to assess for any pathology. Using the information gathered with the PoCUS simulator, the participant would then formulate a revised differential, again with a maximum of five diagnoses and a corresponding percentage confidence for each.

This method of testing was repeated for each participant until all six scenarios had been completed. Following the final scenario, participants reviewed each case in detail with the instructor. Appendix 2 outlines details of scenario development and technical specifications.

Data collection and measurement

Each participant was observed by one of the study investigators and was assessed for completeness of the clinical examination and image acquisition during PoCUS scans. Scans were defined as complete or incomplete, with specific omissions being recorded.

Written differential diagnoses were collected from participants following each scenario. Each of these sheets contained the participant’s ranked differential diagnosis and percentage confidence before and after PoCUS. These differential diagnoses were analysed to assess the change in diagnostic accuracy, confidence, and precision.

Diagnostic accuracy was determined by scoring the ranking of the correct diagnosis on the participant’s differential. For example, if the participant’s first-ranked diagnosis was the correct one, he or she received five points; if it was second-ranked, the participant received four points, and so on. Diagnostic confidence refers to the change in percentage confidence that the participant assigned to the correct diagnosis before and after PoCUS. Diagnostic precision assessed the difference between the pre- and post-PoCUS differentials, and whether PoCUS helped narrow the participant’s differential.

Data analysis

All data analyses was completed using R software, version 2.15 (R Foundation for Statistical Computing, Vienna, Austria). For each participant, an improvement in diagnostic accuracy was defined as either a higher score on his or her diagnostic accuracy after simulator usage or a perfect score on both the before and after diagnostic accuracy with an increase in diagnostic confidence. For each participant, an average of the diagnostic scores for the six scenarios before and after PoCUS was taken. A one-sided Wilcoxon signed-rank test was used to examine any change in diagnostic scores following PoCUS.

Similarly, an average of the confidence in diagnoses measures and the number of viable diagnoses were taken before and after PoCUS was determined. The difference in confidence level was tested with a one-sided paired sample t-test. Diagnostic precision was assessed by the difference in the number of viable diagnoses and tested with a one-sided Wilcoxon signed-rank test.

Results

A total of 12 medical learners were recruited to participate in the study (six third- and fourth-year Dalhousie University medical student clerks, and six PGY1–3 emergency medicine and non-emergency medicine resident physicians. Each of the 12 learners completed the necessary preparation before attending the session, completed training, and completed six testing scenarios during the day of the session. There were no indeterminate results, missing responses, or adverse events during the study.

One participant, during one scenario, failed to complete a scan of the lung bases and thus did not complete a full ACES scan. In addition, one participant during one scenario did not complete a satisfactory respiratory examination.

Differential diagnostic ranking

The addition of PoCUS to the standard simulated assessments improved the differential diagnostic ranking during simulated cardiorespiratory cases. The correct diagnosis was placed higher in the differential after PoCUS in a median of four (interquartile range [IQR]=3, 4.5) out of the six scenarios for each participant. Participants were more likely to have a correct primary diagnosis at the top of their differential after PoCUS simulator usage. Prior to use of PoCUS, 45 correct primary diagnoses were made during the simulated scenarios (62.5%, n=72). Following simulated PoCUS, 64 (88.9%) correct primary diagnoses were made. This increase in a correct primary diagnosis was statistically significant (χ2=14, 1df, p=0.0002). Of the 27 incorrect diagnoses before PoCUS, a total of 20 (74.1%) were improved after PoCUS. There were only three occasions where an incorrect diagnosis superseded a correct diagnosis after the addition of PoCUS, and it was on the same scenario for three different participants (i.e., dilated cardiomyopathy).

Diagnostic accuracy

The mean diagnostic accuracy score before PoCUS was 3.8 (median=4, IQR=3,5, 4.1), whereas the mean score after PoCUS was 4.5 (median=4.6, IQR=4.1, 5.0). The median of the post-PoCUS scores was significantly greater than that of the pre-PoCUS scores (W=66, p=0.004) (Figure 3).

Figure 3 Box and whisker plot of participants’ median of diagnostic accuracy before and after PoCUS.

Diagnostic confidence

Similar to diagnostic accuracy, the confidence increased for all participants in at least two of the scenarios, with a median of four scenarios with confidence level increases (IQR=3, 4.5; Figure 4). All participants who had an improvement in their diagnostic accuracy score also had an improvement in their confidence. There were 44 tests where the participant accurately identified the diagnosis both before and after use of PoCUS. Of these, 29 (65.6%) had increased confidence in the diagnosis after PoCUS.

Figure 4 Box and whisker plot of participants’ median diagnostic confidence before and after PoCUS.

The mean confidence in the diagnosis score pre-PoCUS was 52.2 (SD=14.7), whereas the mean post-PoCUS score was 81.7 (SD=9.5), and the difference in these means was statistically significant (t=−7.71, p<0.0001).

Diagnostic precision

Participants were further able to reduce the number of diagnoses in their differential. Participants narrowed their diagnoses to a median of 4.5 times out of 6 scenarios (IQR=3, 5). Two participants narrowed their number of diagnoses in all six scenarios. The median number of diagnoses for each patient pre-PoCUS was 3.5 (IQR=3.0, 3.8) with a median of 2.3 (IQR=1.5, 2.9) diagnoses post-PoCUS. The difference was statistically significant (W=0, p<0.001) (Figure 5).

Figure 5 Box and whisker plot of participants’ median number of viable diagnoses before and after PoCUS.

Discussion

The results in this study demonstrate that the addition of an ACES simulated PoCUS protocol to standardized simulated cardiorespiratory scenarios leads to improved diagnostic accuracy, confidence, and a narrowed differential for medical learners following a brief and focused standardized training protocol. These findings show that PoCUS protocols can provide medical learners with useful diagnostic information in simulated settings and facilitate both rule in and rule out determinations for a number of conditions.

The improvement in diagnostic characteristics with PoCUS during simulated cardiorespiratory scenarios supports the utility and practicality of teaching focused PoCUS protocols with high-fidelity simulators, particularly in medical emergency scenarios. One of the scenarios, a patient with dilated cardiomyopathy, proved difficult for participants. All three occasions where a worse diagnosis was given post-PoCUS usage involved this specific scenario. It is believed that the counterbalancing or the sequencing approach removed participant fatigue as a cause for this. Possible explanations include 1) the CAE Vimedix ultrasound simulator does not present participants with a valid depiction of dilated cardiomyopathy; 2) the training protocol lacked sufficient emphasis on identifying this pathology; or 3) learners were unfamiliar with this pathology and require more experience to familiarize themselves with the typical clinical and PoCUS characteristics of dilated cardiomyopathy.

Only one participant during one scenario performed an inadequate ACES scan by failing to scan the lung bases during the pleural effusion scenario. Because this respiratory pathology has ultrasound findings near the lung bases, the participant was unable to accurately identify and interpret the effusion. Interestingly, both the pre- and post-PoCUS differentials did not include pleural effusion. This example illustrates the importance of completing a thorough PoCUS scan.

The results in this study are consistent with the previously reported benefits of PoCUS protocols in the clinical setting. Jones et al. reported the diagnostic benefit of PoCUS in patients presenting to the emergency department with undifferentiated hypotension 4 and found that a PoCUS protocol decreased the median number of viable diagnoses (from four to three median diagnoses per differential) and increased the proportion of differentials with the correct diagnosis ranked first (from 50% to 80%). Similarly, it was found that a simulated PoCUS protocol decreased the median number of viable diagnoses by 1.2 (from 3.5 to 2.3) and increased the proportion of differentials with a correct primary diagnosis by 26.4% (from 62.5% to 88.9%).

Limitations

Because USIM was limited entirely to simulation, no clinical conclusions can be drawn from these findings. In terms of the realism of the simulated scenarios, the protocol did not include treatment options. The high-fidelity mannequin simulator is capable of responding to treatment, and it could have been made more realistic if participants were given the option of treating the simulated patient. Appropriate responses and clinical improvement or deterioration may provide participants with an increased feeling that a scenario is realistic and an increased opportunity to hone treatment skills.

Study participants were medical student clerks and residents with varying degrees of experience and knowledge. Given the small number of participants, their heterogeneity, and the fact that the study was conducted at a single centre, the external validity of the results may be limited.

While unlikely to fully explain the findings, it is possible that the improved diagnostic performance could be due to participants having more time to reconsider the case, as they performed the ultrasound after their initial assessment. The diagnostic accuracy score has not been validated elsewhere.

As previously discussed, there are limitations to the effectiveness of high-fidelity simulation training, primarily due to shortcomings of simulation technologies. Although the high-fidelity simulators employed in this study were state of the art, it is unlikely that these simulated scenarios exactly mirrored real-life events and suspended participant disbelief.

In addition, the current study did not focus on the translation of diagnostic benefits into a clinical setting or the long-term retention of PoCUS competency.

Future Research and Follow-Up

The participants will be followed to determine the extent of their long-term retention of this training and knowledge, particularly in terms of their ongoing training and clinical implementation of PoCUS. Future research is required to determine whether the effects found will translate from the simulation setting into the real-life clinical setting.

Conclusions

This pilot study suggests that for medical learners newly competent in PoCUS, the addition of an ACES PoCUS protocol to standard clinical assessment improves diagnostic accuracy, confidence, and precision during simulated cardiorespiratory scenarios. This is consistent with clinical studies and supports the use of ultrasound during medical simulation.

Supplementary material

To view supplementary material for this article, please visit http://dx.doi.org/10.1017/cem.2014.41.

References

1. Frederiksen, CA, Juhl-Olsen, P, Sloth, E. Advances in imaging: ultrasound in every physician’s pocket. Expert Opin Med Diagn 2012;6:167-170.
2. Moore, CL, Copel, JA. Point-of-care ultrasonography. N Engl J Med 2011;364(8):749-757.
3. Volpicelli, G, Cardinale, L, Berchialla, P, et al. A comparison of different diagnostic tests in the bedside evaluation of pleuritic pain in the ED. Am J Emerg Med 2012;30:317-324.
4. Jones, AE, Tayal, VS, Sullivan, DM, et al. Randomized, controlled trial of immediate versus delayed goal-directed ultrasound to identify the cause of nontraumatic hypotension in emergency department patients. Crit Care Med 2004;32(8):1703-1708.
5. Plummer, D, Brunette, D, Asinger, R, et al. Emergency department echocardiography improves outcome in penetrating cardiac injury. Ann Emerg Med 1992;21:709-712.
6. Mandavia, DP, Hoffner, RJ, Mahaney, K, Henderson, SO. Bedside echocardiography by emergency physicians. Ann Emerg Med 2001;38:377-382.
7. Volpicelli, G, Elbarbary, M, Blaivas, M, et al. International Liaison Committee on Lung Ultrasound (ILC-LUS) for International Consensus Conference on Lung Ultrasound (ICC-LUS) (2012) International evidence-based recommendations for point-of-care lung ultrasound. Intensive Care Med 2012;38(4):577-591.
8. Hwang, JQ, Kimberly, HH, Liteplo, AS, et al. An evidence-based approach to emergency ultrasound. Emerg Med Pract 2011;13:1-27; quiz 27-8.
9. Gogalniceanu, P, Sheena, Y, Kashef, E, et al. Is basic emergency ultrasound training feasible as part of standard undergraduate medical education? J Surg Educ 2010;67:152-156.
10. Chin, EJ, Chan, CH, Mortazavi, R, et al. A pilot study examining the viability of a prehospital assessment with ultrasound for emergencies (PAUSE) protocol. J Emerg Med 2013;44:142-149.
11. Atkinson, PRT, McAuley, DJ, Kendall, RJ, et al. Abdominal and Cardiac Evaluation with Sonography in Shock (ACES): an approach by emergency physicians for the use of ultrasound in patients with undifferentiated hypotension. Emerg Med J 2009;26:87-91.
12. Parks, AR, Atkinson, P, Verheul, G, et al. Can medical learners achieve point-of-care ultrasound competency using a high-fidelity ultrasound simulator?: a pilot study. Crit Ultrasound J 2013;5:9.
13. Blum, T, Rieger, A, Navab, N, et al. A review of computer-based simulators for ultrasound training. Simul Healthc 2013;8:98-108.
14. Simel, DL, Rennie, D, Keitz, SA. The rational clinical examination: evidence-based clinical diagnosis. New York: McGraw-Hill Medical; [Chicago, IL]: The JAMA Network Journals, American Medical Association, 2009. 744 pgs ISBN: 9780071590303.
15. Family Practice Notebook . Available at: http://www.fpnotebook.com/ (accessed March 2012).