Skip to main content Accessibility help
×
Home

A transparent and defensible process for applicant selection within a Canadian emergency medicine residency program

  • Quinten S. Paterson (a1), Riley Hartmann (a1), Rob Woods (a1), Lynsey J. Martin (a1) and Brent Thoma (a1)...

Abstract

Objectives

The Canadian Resident Matching Service (CaRMS) selection process has come under scrutiny due to the increasing number of unmatched medical graduates. In response, we outline our residency program's selection process including how we have incorporated best practices and novel techniques.

Methods

We selected file reviewers and interviewers to mitigate gender bias and increase diversity. Four residents and two attending physicians rated each file using a standardized, cloud-based file review template to allow simultaneous rating. We interviewed applicants using four standardized stations with two or three interviewers per station. We used heat maps to review rating discrepancies and eliminated rating variance using Z-scores. The number of person-hours that we required to conduct our selection process was quantified and the process outcomes were described statistically and graphically.

Results

We received between 75 and 90 CaRMS applications during each application cycle between 2017 and 2019. Our overall process required 320 person-hours annually, excluding attendance at the social events and administrative assistant duties. Our preliminary interview and rank lists were developed using weighted Z-scores and modified through an organized discussion informed by heat mapped data. The difference between the Z-scores of applicants surrounding the interview invitation threshold was 0.18-0.3 standard deviations. Interview performance significantly impacted the final rank list.

Conclusions

We describe a rigorous resident selection process for our emergency medicine training program which incorporated simultaneous cloud-based rating, Z-scores, and heat maps. This standardized approach could inform other programs looking to adopt a rigorous selection process while providing applicants guidance and reassurance of a fair assessment.

RésuméObjectif

Le processus de sélection des résidents au Canada a été soumis à un examen en raison du nombre croissant de diplômés non jumelés en médecine. Aussi avons-nous décidé d'exposer, dans les grandes lignes, un processus de sélection des candidats à un programme de résidence, ainsi que la manière dont ont été intégrées des pratiques exemplaires et des techniques novatrices.

Méthode

Des réviseurs de dossiers et des intervieweurs ont été choisis afin d'atténuer les préjugés sexistes et d'accroître la diversité de l’équipe d’évaluation. Quatre résidents et deux médecins traitants ont évalué chacun des dossiers à l'aide d'un modèle de revue commun, sauvegardé dans le nuage afin de permettre des évaluations simultanées. De leur côté, les candidats ont passé un entretien dans quatre salles de rencontre montées pareillement et comptant deux ou trois intervieweurs chacune. Nous avons utilisé des cartes de couleurs pour examiner les différences d’évaluation et éliminer les variations à l'aide de la technique des écarts réduits Z. Le nombre d'heures-personnes nécessaires pour mener à terme le processus de sélection a été quantifié, et les résultats sont présentés sous forme de statistiques et de graphiques.

Résultats

Nous avons reçu de 75 à 90 demandes de jumelage du CaRMS au cours de chacun des cycles, entre 2017 et 2019. Le processus global de sélection a nécessité 320 heures-personnes sur une base annuelle, outre la participation à des activités sociales et les responsabilités d'adjoint administratif. Les listes préliminaires d'entrevue et de classement ont été élaborées à l'aide des écarts Z pondérés, puis modifiées dans le cadre d'une discussion structurée et guidée par les cartes de couleurs. La différence entre les écarts réduits Z des candidats relativement au seuil d'invitation à une entrevue était de 0,18-0,3 en écart type. La performance des candidats à l'entrevue a eu une forte incidence sur la liste finale de classement.

Conclusions

A été exposé, dans l'article, un processus rigoureux de sélection des résidents à un programme de formation en médecine d'urgence, intégrant à la fois un modèle d’évaluation infonuagique, des formules de calcul d’écarts réduits Z et des cartes de couleurs. Cette approche structurée pourrait à la fois servir de guide à des responsables d'autres programmes à la recherche d'un processus rigoureux de sélection, et donner aux candidats une direction à suivre et l'assurance d'une évaluation juste.

Copyright

Corresponding author

Correspondence to: Dr. Quinten Paterson, PGY-4 FRCPC Emergency Medicine, Department of Emergency Medicine, College of Medicine, University of Saskatchewan, Room 2646, 103 Hospital Drive, Saskatoon, SKS7N 0W8; Email: quintenpaterson@gmail.com

References

Hide All
1.Ryan, T.Addressing bias and lack of objectivity in the Canadian resident matching process. CMAJ 2018;190:E1211–2.
2.Apramian, T, Ramazani, F, Lee, D, et al. Position paper: Support for unmatched Canadian medical students. 2017. Available at: https://www.cfms.org/files/position-papers/agm_2017_support_unmatched.pdf (accessed January 13, 2019).
3.Owens, B. If Canada needs more doctors, why hasn't medical school enrolment increased? CMAJ News 2018. Available at: https://cmajnews.com/2018/10/03/if-canada-needs-more-doctors-why-hasnt-medical-school-enrolment-increased-cmaj-109-5649/ (accessed January 13, 2019).
4.Vogel, L.Record number of unmatched medical graduates. CMAJ 2017;189(21):E758–9.
5.AFMC Board of Directors. Reducing the number of unmatched Canadian medical graduates: A way forward 2018. Available at: https://afmc.ca/sites/default/files/pdf/AFMC_reportreducingunmatchedcdnmg_EN.pdf (January 15, 2019).
6.Persad, A.The unmatched. CMAJ 2018;9(2):e89–92.
7.Persad, A.The overall culture of residency selection needs fixing. CMAJ 2018;190(14):E443.
8.Bandiera, G, Wycliffe-Jones, K, Busing, N, et al. Resident selection in Canada: What do program directors think about best practice recommendations? Poster presented at: Family Medicine Forum; 2015 Nov 12-14; Toronto, ON.
9.McInnes, M.Residency matching woes. CMAJ 2015;187(5):357.
10.Bandiera, G, Abrahams, C, Cipolla, A, et al. Best practices in applications & selection; 2013. Available at: https://pg.postmd.utoronto.ca/wp-content/uploads/2016/06/BPASDraftFinalReportPGMEACMay2013.pdf (accessed December 6, 2018).
11.Bandiera, G, Abrahams, C, Ruetalo, M, et al. Identifying and promoting best practices in residency application and selection in a complex academic health network. Acad Med 2015;90(12):1594–601.
12.Ben-Nun, P.Respondent fatigue. In: Lavrakas, PJ, editor. Encyclopedia of Survey Research Methods. Thousand Oaks, CA: Sage Publications, Inc.; 2011. p. 743. Available at: http://methods.sagepub.com/reference/encyclopedia-of-survey-research-methods/n480.xml (accessed February 25, 2019).
13.DeSantis, M, Marco, CA.Emergency medicine residency selection: factors influencing candidate decisions. Acad Emerg Med 2005;12(6):559–61.
14.Love, JN, Howell, JM, Hegarty, CB, et al. Factors that influence medical student selection of an emergency medicine residency program: implications for training programs. Acad Emerg Med 2012;19(4):455–60.
15.Thurman, RJ, Katz, E, Carter, W, et al. Emergency medicine residency applicant perceptions of unethical recruiting practices and illegal questioning in the match. Acad Emerg Med 2009;16(6):550–7.
16.Hayden, SR, Hayden, M, Gamst, A.What characteristics of applicants to emergency medicine residency programs predict future success as an emergency medicine resident? Acad Emerg Med 2005;12(3):206–10.
17.Bhat, R, Takenaka, K, Levine, B, et al. Predictors of a top performer during emergency medicine residency. J Emerg Med 2015;49(4):505–12.
18.Borowitz, SM, Saulsbury, FT, Wilson, WG.Information collected during the residency match process does not predict clinical performance. Arch Pediatr Adolesc Med 2000;154(3):256–60.
19.Hopson, LR, Burkhardt, JC, Stansfield, RB, et al. The multiple mini-interview for emergency medicine resident selection. J Emerg Med 2014;46(4):537–43.
20.Oyama, LC, Kwon, M, Fernandez, JA, et al. Inaccuracy of the global assessment score in the emergency medicine standard letter of recommendation. Acad Emerg Med 2010;17(2Suppl 2):S38–41.
21.Dore, KL, Kreuger, S, Ladhani, M, et al. The reliability and acceptability of the Multiple Mini-Interview as a selection instrument for postgraduate admissions. Acad Med 2010;85(10 Suppl):S60–3.
22.Boursicot, K, Etheridge, L, Setna, Z, et al. Performance in assessment: consensus statement and recommendations from the Ottawa conference. Med Teach 2011;33(5):370–83.
23.Gormley, G.Summative OSCEs in undergraduate medical education. Ulster Med J 2011;80(3):127–32.
24.Pau, A, Jeevaratnam, K, Chen, YS, et al. The Multiple Mini-Interview (MMI) for student selection in health professions training - a systematic review. Med Teach 2013;35(12):1027–41.
25.Bandiera, G, Regehr, G.Reliability of a structured interview scoring instrument for a Canadian postgraduate emergency medicine training program. Acad Emerg Med 2004;11(1):2732.
26.Eneh, AA, Jagan, L, Baxter, S.Relative importance of the components of the Canadian Residency Matching Service application. Can J Ophthalmol 2014;49(5):407–13.
27.Gorouhi, F, Alikhan, A, Rezaei, A, Fazel, N.Dermatology residency selection criteria with an emphasis on program characteristics: A national program director survey. Dermatol Res Pract 2014;2014:692760.
28.Kenny, S, McInnes, M, Singh, V.Associations between residency selection strategies and doctor performance: a meta-analysis. Med Educ 2013;47(8):790800.
29.McCann, SD, Nomura, JT, Terzian, WT, Breyer, MJ, Davis, BJ.Importance of the emergency medicine application components: the medical student perception. J Emerg Med 2016;50(3):466–70.e.1.
30.Cohen, JS, Patten, S.Well-being in residency training: a survey examining resident physician satisfaction both within and outside of residency training and mental health in Alberta. BMC Med Educ 2005;5(21):21.
31.Raj, KS.Well-being in residency: A systematic review. J Grad Med Educ 2016;8(5):674–84.
32.Feldman, M, Lazzara, EH, Vanderbilt, AA, DiazGranados, D.Rater training to support high-stakes simulation-based assessments. J Contin Educ Health Prof 2012;32(4):279–86.
33.Woehr, DJ, Huffcutt, AI.Rater training for performance appraisal: A quantitative review. J Occup Organ Psychol 1994;67(3):189205.
34.Kreiter, CD.A proposal for evaluating the validity of holistic-based admission processes. Teach Learn Med 2013;25(1):s–7.
35.Witzburg, RA, Sondheimer, HM.Holistic review—shaping the medical profession one applicant at a time. N Engl J Med 2013;368(17):1565–7.

Keywords

Type Description Title
UNKNOWN
Supplementary materials

Paterson et al. supplementary material
Paterson et al. supplementary material

 Unknown (142 KB)
142 KB

A transparent and defensible process for applicant selection within a Canadian emergency medicine residency program

  • Quinten S. Paterson (a1), Riley Hartmann (a1), Rob Woods (a1), Lynsey J. Martin (a1) and Brent Thoma (a1)...

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed