Introduction: In 2018, Canadian postgraduate specialist Emergency Medicine (EM) programs began implementing a competency-based medical education (CBME) assessment system. To support improvement of this assessment program, we sought to evaluate its short-term educational outcomes nationally and within individual programs. Methods: Program-level data from the 2018 resident cohort were amalgamated and analyzed. The number of Entrustable Professional Activity (EPA) assessments (overall and for each EPA) and the timing of resident promotion through program stages was compared between programs and to the guidelines provided by the national EM specialty committee. Total EPA observations from each program were correlated with the number of EM and pediatric EM rotations. Results: Data from 15 of 17 (88.2%) EM programs containing 9,842 EPA observations from 68 of the 77 (88.3%) Canadian EM specialist residents in the 2018 cohort were analyzed. The average number of EPAs observed per resident in each program varied from 92.5 to 229.6 and correlated strongly with the number of blocks spent on EM and pediatric EM (r = 0.83, p < 0.001). Relative to the guidelines outlined by the specialty committee, residents were promoted later than expected and with fewer EPA observations than suggested. Conclusion: We present a new approach to the amalgamation of national and program-level assessment data. There was demonstrable variation in both EPA-based assessment numbers and promotion timelines between programs and with national guidelines. This evaluation data will inform the revision of local programs and national guidelines and serve as a starting point for further reaching outcome evaluation. This process could be replicated by other national assessment programs.

Introduction: The oral case presentation is recognized as a core educational and patient care activity but has not been well studied in the emergency setting. The objectives of this study are: 1) to develop a competency-based assessment tool to formally evaluate the emergency medicine oral case presentation (EM-OCP) competency of medical students and ‘transition to discipline’ residents, and 2) to develop, implement and evaluate a curriculum to enhance oral case presentation (OCP) communication skills in the emergency medicine (EM) setting. Methods: Using data from a literature review, a Canadian Association of Emergency Physicians national survey, and local focus groups, the authors designed an OCP framework, blended learning curriculum, and EM-OCP assessment tool. Ninety-six clerkship students were randomly assigned to receive either the control, the standard clerkship curriculum, or intervention, the blended learning curriculum. At the beginning of their emergency medicine rotation, learners completed a pre-test using a standardized patient (SP) case to assess their baseline OCP skills. The intervention group then completed the EM-OCP curriculum. All students completed post-tests with a different SP at the end of the six-week EM rotation. Audio-recordings of pre and post-tests were evaluated using the assessment tool by two blinded evaluators. Results: Using the Kruskal-Wallis test, all students demonstrated improvement in EM-OCP skills between their pre-test and post-test, however, those who received the blended learning curriculum showed significantly greater improvement in synthesis of information (p = 0.044), management (p = 0.006) and overall entrustment decision score (p = 0.000). Conclusion: Implementation of a novel EM-OCP curriculum resulted in more effective communication and higher entrustment scores. This curriculum could improve OCP performance not only in emergency medicine settings but also across specialties where medical students and residents must manage critical patients.

Introduction: With the transition of Emergency Medicine into competency based medical education (CBME), entrustable professional activities (EPAs) are used to evaluate residents on performed clinical duties. This study aimed to determine if implementing a case-based orientation, designed to increase recognition of available EPAs, into CBME orientation would help residents increase the number of EPAs completed. Methods: We designed an intervention consisting of clinical cases that were reviewed by national EPA experts who identified which EPAs could be assessed from each case. A case-based session was incorporated into the 2019 CBME orientation for The McMaster Emergency Medicine Program. Postgraduate Year (PGY)1 residents read the cases and discussed which EPAs could be obtained with PGY2/faculty facilitators. The number of EPAs completed in the first two blocks of PGY1 was determined from local program data and Student's t-test was used to compare averages between cohorts. Results: We analyzed data from 22 trainees (7 in 2017, 8 in 2018, and 7 in 2019). In the first two blocks of PGY1, the intervention cohort (2019) had a significantly higher average number of EPAs completed per trainee (47.4 [SD 11.8]) than the historical cohort (25.3 [SD 6.7]) (p < 0.001) (Cohen's d = 2.3). No significant difference existed in the number EPAs obtained between the 2017/2018 cohorts, with averages of 24.3 [SD 6.8] and 26.1 [SD 7.0] per trainee respectively (p = 0.6). Conclusion: Implementation of a case-based orientation led by CBME-experienced facilitators nearly doubled the EPA acquisition rate of our PGY1s. The consistent EPA acquisition by the 2017/2018 cohorts suggest that the post-intervention increase was not solely due to developed familiarity with the CBME curriculum.

Introduction: For many years, Emergency Medicine (EM) educators have used narrative comments to assess their learners on each shift, either in isolation or combined with some type of Likert scale ranking. Competency based medical education (CBME), soon to be fully implemented throughout Canadian EM educational programs, encourages this type of frequent low-stakes narrative assessment. It is important to understand what information is currently garnered from existing narrative assessments in order to successfully and smoothly transition to the CBME system. The purpose of this study was to explore how one Canadian undergraduate EM program's narrative assessment comments mapped to two competency frameworks: one traditional CanMEDS-based and one competency-based, built on entrustable professional activities (EPAs). Methods: A qualitative and quantitative content analysis of 1925 retrospective, narrative assessments was conducted for the 2015/2016 and 2016/2017 academic years. The unprompted comments were mapped to the Royal College CanMEDS framework and the Association of Faculties of Medicine of Canada EPA Framework. Using an iterative coding process as per accepted qualitative methodologies, additional codes were generated to classify comments and identify themes that were not captured by either framework. Results: 93% and 85% of the unprompted narrative assessments contained comments that mapped to at least one CanMEDS role or EPA competency, respectively. The most common CanMEDS role commented upon was Medical Expert (86%), followed by Communicator, Collaborator and Scholar (all at 23%). The most common EPA competency mentioned related to history and physical findings (62%) followed by management plan (33%), and differential diagnosis (33%). However, 75% of narrative comments contained within the assessments, included ideas that did not fall into either framework but were repeated with frequency to suggest importance. The experiential characteristics of working with a learner were commented upon by 22% of preceptors. Other unmapped themes included contextual information, generalities and platitudes, and directed feedback for next steps to improve. Conclusion: While much of the currently captured data can be mapped to established frameworks, important information for both learner and assessor may be lost by limiting comments to the competencies described within a particular framework, suggesting caution when transitioning to a CBME assessment program.