Hostname: page-component-cd9895bd7-gvvz8 Total loading time: 0 Render date: 2024-12-27T15:46:32.320Z Has data issue: false hasContentIssue false
  • English
  • Français

On the Assessment of Paramedic Competence: A Narrative Review with Practice Implications

Published online by Cambridge University Press:  30 November 2015

W. Tavares  and
S. Boet
W. Tavares*
Affiliation:
Centennial College, Paramedic Program, Toronto, Ontario, Canada McMaster University, Faculty of Medicine, Division of Emergency Medicine, Hamilton, Ontario, Canada ORNGE Transport Medicine, Mississauga, Ontario, Canada York Region Paramedic Services, Newmarket, Ontario, Canada Paramedic Association of Canada, Ottawa, Ontario, Canada
S. Boet
Affiliation:
University of Ottawa, Skills and Simulation Centre and Department of Anesthesiology, Ottawa Hospital, Ottawa, Ontario, Canada
*
Correspondence: Walter Tavares, PhD Centennial College P.O. Box 631, Station A Toronto, Ontario, Canada M1K 5E9 E-mail: wtavares@centennialcollege.ca
Get access Rights & Permissions [Opens in a new window]

Abstract

Introduction

Paramedicine is experiencing significant growth in scope of practice, autonomy, and role in the health care system. Despite clinical governance models, the degree to which paramedicine ultimately can be safe and effective will be dependent on the individuals the profession deems suited to practice. This creates an imperative for those responsible for these decisions to ensure that assessments of paramedic competence are indeed accurate, trustworthy, and defensible.

Purpose

The purpose of this study was to explore and synthesize relevant theoretical foundations and literature informing best practices in performance-based assessment (PBA) of competence, as it might be applied to paramedicine, for design or evaluation of assessment programs.

Methods

A narrative review methodology was applied to focus intentionally, but broadly, on purpose relevant, theoretically derived research that could inform assessment protocols in paramedicine. Primary and secondary studies from a number of health professions that contributed to and informed best practices related to the assessment of paramedic clinical competence were included and synthesized.

Results

Multiple conceptual frameworks, psychometric requirements, and emerging lines of research are forwarded. Seventeen practice implications are derived to promote understanding as well as best practices and evaluation criteria for educators, employers, and/or licensing/certifying bodies when considering the assessment of paramedic competence.

Conclusions

The assessment of paramedic competence is a complex process requiring an understanding, appreciation for, and integration of conceptual and psychometric principles. The field of PBA is advancing rapidly with numerous opportunities for research.

TavaresW , BoetS . On the Assessment of Paramedic Competence: A Narrative Review with Practice Implications. Prehosp Disaster Med. 2016;31(1):64–73.

Keywords

clinical competenceeducational measurementEmergency Medical ServicesEmergency Medical Technicians
Type
Comprehensive Review
Information
Prehospital and Disaster Medicine , Volume 31 , Issue 1 , February 2016 , pp. 64 - 73
Copyright
© World Association for Disaster and Emergency Medicine 2015 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

1. Atack, L, Maher, J. Emergency medical and health providers’ perceptions of key issues in prehospital patient safety. Prehosp Emerg Care. 2009;14(1):95-102.CrossRefGoogle Scholar
2. Bigham, BL, Buick, JE, Brooks, SC, Morrison, M, Shojania, KG, Morrison, LJ. Patient safety in Emergency Medical Services: a systematic review of the literature. Prehosp Emerg Care. 2012;16(1):20-35.CrossRefGoogle ScholarPubMed
3. Fairbanks, RJ, Crittenden, CN, O’Gara, KG, et al. Emergency Medical Services provider perceptions of the nature of adverse events and near‐misses in out‐of‐hospital care: an ethnographic view. Acad Med. 2008;15(7):633-640.Google ScholarPubMed
4. Jensen, J, Croskerry, P, Travers, A. Consensus on paramedic clinical decisions during high-acuity emergency calls: results of a Canadian Delphi study. Can J Emerg Med. 2011;13(5):310-318.CrossRefGoogle ScholarPubMed
5. Van der Vleuten, C, Schuwirth, L. Assessing professional competence: from methods to programs. Med Educ. 2005;39(3):309-317.CrossRefGoogle Scholar
6. Miller, G. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9):S63-S67.CrossRefGoogle ScholarPubMed
7. Holmboe, E, Sherbino, J, Long, DM, Swing, SR, Frank, JR. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676-682.CrossRefGoogle ScholarPubMed
8. Regener, H. A proposal for student assessment in paramedic education. Med Teach. 2005;27(3):234-241.CrossRefGoogle ScholarPubMed
9. Su, E, Schmidt, TA, Mann, C, Zechnich, AD. A randomized controlled trial to assess decay in acquired knowledge among paramedics completing a pediatric resuscitation course. Acad Emerg Med. 2000;7(7):779-786.CrossRefGoogle ScholarPubMed
10. Lammers, RL, Byrwa, MJ, Fales, WD, Hale, RA. Simulation-based assessment of paramedic pediatric resuscitation skills. Prehosp Emerg Care. 2009;13(3):345-356.CrossRefGoogle ScholarPubMed
11. Studnek, JR, Fernandez, AR, Shimberg, B, Garifo, M, Correll, M. The association between Emergency Medical Services field performance assessed by high‐fidelity simulation and the cognitive knowledge of practicing paramedics. Acad Emerg Med. 2011;18(11):1177-1185.CrossRefGoogle ScholarPubMed
12. Unlüer, EE, Yavasi, O, Kara, PH, et al. Paramedic-performed Focused Assessment with Sonography in Trauma (FAST) in the emergency department. Ulus Travma Acil Cerrahi Derg. [Turkish Journal of Trauma & Emergency Surgery: TJTES]. 2011;17(2):113-116.CrossRefGoogle ScholarPubMed
13. Sahni, R, Menegazzi, JJ, Mosesso, VN. Paramedic evaluation of clinical indicators of cervical spinal injury. Prehosp Emerg Care. 1997;1(1):16-18.CrossRefGoogle ScholarPubMed
14. Zautcke, JL, Lee, RW, Ethington, NA. Paramedic skill decay. J Emerg Med. 1987;5(6):505-512.CrossRefGoogle ScholarPubMed
15. LeBlanc, V, MacDonald, RD, McArthur, B, King, K, Lepine, T. Paramedic performance in calculating drug dosages following stressful scenarios in a human patient simulator. Prehosp Emerg Care. 2005;9(4):439-444.CrossRefGoogle Scholar
16. Hubble, MW, Paschal, KR, Sanders, TA. Medication calculation skills of practicing paramedics. Prehosp Emerg Care. 2000;4(3):253-260.CrossRefGoogle ScholarPubMed
17. Eastwood, KJ, Boyle, MJ, Williams, B. Paramedics’ ability to perform drug calculations. West J Emerg Med. 2009;10(4):240-243.Google ScholarPubMed
18. Rocca, B, Crosby, E, Maloney, J, Bryson, G. An assessment of paramedic performance during invasive airway management. Prehosp Emerg Care. 2000;4(2):164-167.CrossRefGoogle ScholarPubMed
19. Ruetzler, K, Roessler, B, Potura, L, et al. Performance and skill retention of intubation by paramedics using seven different airway devices—a manikin study. Resuscitation. 2011;82(5):593-597.CrossRefGoogle ScholarPubMed
20. Manser, T, Foster, S, Gisin, S, Jaeckel, D, Ummenhofer, W. Assessing the quality of patient handoffs at care transitions. Qual Saf Health Care. 2010;19(6):e44-e44.Google ScholarPubMed
21. LeBlanc, VR, Regehr, C, Tavares, W, Scott, AK, MacDonald, R, King, K. The impact of stress on paramedic performance during simulated critical events. Prehosp Disaster Med. 2012;27(4):369-374.CrossRefGoogle ScholarPubMed
22. Gordon, DL, Issenberg, SB, Gordon, MS, LaCombe, D, McGaghi, WC, Petrusa, ER. Stroke training of prehospital providers: an example of simulation-enhanced blended learning and evaluation. Med Teach. 2005;27(2):114-121.CrossRefGoogle ScholarPubMed
23. Losek, JD, Szewczuga, D, Glaeser, PW. Improved prehospital pediatric ALS care after an EMT-paramedic clinical training course. Am J Emerg Med. 1994;12(4):429-432.CrossRefGoogle ScholarPubMed
24. Pointer, JE. Clinical characteristics of paramedics’ performance of pediatric endotracheal intubation. Am J Emerg Med. 1989;7(4):364-366.CrossRefGoogle ScholarPubMed
25. Lee, KHK, Grantham, H, Boyd, R. Comparison of high‐and low‐fidelity mannequins for clinical performance assessment. Emerg Med (Fermantle). 2008;20(6):508-514.Google ScholarPubMed
26. Wyatt, A, Fallows, B, Archer, F. Do clinical simulations using a human patient simulator in the education of paramedics in trauma care reduce error rates in preclinical performance? Prehosp Emerg Care. 2004;8(4):435-436.CrossRefGoogle Scholar
27. Castle, N, Owen, R, Hann, M, Naidoo, R, Reeves, D. Assessment of the speed and ease of insertion of three supraglottic airway devices by paramedics: a manikin study. Emerg Med J. 2010;27(11):860-863.CrossRefGoogle ScholarPubMed
28. Hein, C, Owen, H, Plummer, J. A training program for novice paramedics provides initial laryngeal mask airway insertion skill and improves skill retention at 6 months. Simul Healthc. 2010;5(1):33-39.CrossRefGoogle ScholarPubMed
29. Cook, DA, Hatala, R, Brydges, R, et al. Technology-enhanced simulation for health professions education. JAMA. 2011;306(9):978-988.CrossRefGoogle ScholarPubMed
30. Kogan, JR, Holmboe, ES, Hauer, KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009;302(12):1316-1326.CrossRefGoogle ScholarPubMed
31. Martin, M, Hubble, MW, Hollis, M, Richards, ME. Inter-evaluator reliability of a mock paramedic practical examination. Prehosp Emerg Care. 2012;16(2):277-283.CrossRefGoogle Scholar
32. Janing, J, Sime, WR. Inter-rater reliability of paramedic student field performance evaluations. Prehosp Emerg Care. 1999;3(3):265-266.CrossRefGoogle Scholar
33. Tavares, W, LeBlanc, VR, Mausz, J, Sun, V, Eva, KW. Simulation-based assessment of paramedics and performance in real clinical contexts. Prehosp Emerg Care. 2014;18(1):116-122.CrossRefGoogle ScholarPubMed
34. Tavares, W, Boet, S, Theriault, R, Mallette, T, Eva, KW. Global rating scale for the assessment of paramedic clinical competence. Prehosp Emerg Care. 2012;17(1):57-67.CrossRefGoogle ScholarPubMed
35. Van der Vleuten, CPM. The assessment of professional competence: developments, research and practical implications. Adv Health Sci Edu Theory Pract. 1996;1(1):41-67.CrossRefGoogle ScholarPubMed
36. Van der Vleuten, C, Schuwirth, LW, Scheele, F, Driessen, EW, Hodges, B. The assessment of professional competence: building blocks for theory development. Best Pract Res Clin Obstet Gynaecol. 2010;24(6):703-719.CrossRefGoogle ScholarPubMed
37. Boulet, JR, Murray, DJ. Simulation-based assessment in anesthesiology: requirements for practical implementation. Anesthesiology. 2010;112(4):1041-1052.CrossRefGoogle Scholar
38. Boulet, JR, Murray, D. Review article: assessment in anesthesiology education. Can J Anaesth. 2012;59(2):182-192.CrossRefGoogle ScholarPubMed
39. Hodges, B. OSCE! Variations on a theme by Harden. Med Educ. 2003;37(12):1134-1140.CrossRefGoogle ScholarPubMed
40. Hodges, B. Validity and the OSCE. Med Teach. 2003;25(3):250-254.CrossRefGoogle ScholarPubMed
41. Hodges, BD. The objective structured clinical examination: three decades of development. J Vet Med Educ. 2006;33(4):571-577.CrossRefGoogle ScholarPubMed
42. Norcini, JJ, McKinley, DW. Assessment methods in medical education. Teach Teach Educ. 2007;23(3):239-250.CrossRefGoogle Scholar
43. Norcini, J, Anderson, B, Bollela, V, et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 2011;33(3):206-214.CrossRefGoogle ScholarPubMed
44. Wass, V, Van der Vleuten, C, Shatzer, J, Jones, R. Assessment of clinical competence. Lancet. 2001;357(9260):945-949.CrossRefGoogle ScholarPubMed
45. Epstein, R, Hundert, E. Defining and assessing professional competence. JAMA. 2002;287(2):226-235.CrossRefGoogle ScholarPubMed
46. Epstein, R. Assessment in medical education. N Engl J Med. 2007;356(4):387-396.CrossRefGoogle ScholarPubMed
47. Williams, R, Klamen, D, McGaghie, W. Special Article: cognitive, social, and environmental sources of bias in clinical performance ratings. Teach Learn Med. 2003;15(4):270-292.CrossRefGoogle ScholarPubMed
48. Swanson, D, Norman, G, Linn, R. Performance-based assessment: lessons from the health professions. Educ Res. 1995;24(5):5.CrossRefGoogle Scholar
49. Regehr, G, Eva, K, Ginsburg, S, Halwani, Y, Sidhu, R. Assessment in postgraduate medical education: trends and issues in assessment in the workplace. The Future of Medical Education in Canada. 2011.Google Scholar
50. Schuwirth, L, Southgate, L, Page, GG, et al. When enough is enough: a conceptual basis for fair and defensible practice performance assessment. Med Educ. 2002;36(10):925-930.CrossRefGoogle ScholarPubMed
51. Kane, MT. The assessment of professional competence. Eval Health Prof. 1992;15(2):163-182.CrossRefGoogle ScholarPubMed
52. Frank, JR, Mungroo, R, Ahmad, Y, Wang, M, DeRossi, S, Horsley, T. Toward a definition of competency-based education in medicine: a systematic review of published definitions. Med Teach. 2010;32(8):631-637.CrossRefGoogle Scholar
53. Tavares, W, Mausz, J. Assessment of non-clinical attributes in paramedicine using multiple mini-interviews. Emerg Med J. 2013;32(1):70-75.CrossRefGoogle ScholarPubMed
54. Tavares, W, et al. Simulation-based assessment of paramedics and performance in real clinical contexts. Prehosp Emerg Care. 2014;18(1):116-122.CrossRefGoogle ScholarPubMed
55. Brennan, RL. Generalizability Theory. Heidelberg, Berlin: Springer Verlag; 2001.CrossRefGoogle Scholar
56. Brennan, RL. Educational Measurement. Westport, Connecticut USA: Praeger Pub Text; 2006.Google Scholar
57. Crossley, J, Jolly, B. Making sense of work‐based assessment: ask the right questions, in the right way, about the right things, of the right people. Med Educ. 2012;46(1):28-37.CrossRefGoogle ScholarPubMed
58. Norcini, JJ. Current perspectives in assessment: the assessment of performance at work. Md Educ. 2005;39(9):880-889.Google ScholarPubMed
59. Boulet, J, Swanson, D. Psychometric challenges of using simulations for high-stakes assessment. Simulations in Critical Care Education and Beyond. Des Plains, Illinois USA: Society of Critical Care Medicine; 2004: 119-130.Google Scholar
60. Eva, K. “Assessment Strategies in Medical Education.” In: Salerno-Kennedy, (ed). Medical Education: State of the Art. Halifax, Nova Scotia, Canada: Nova Scotia Publishers, Inc.; 2010.Google Scholar
61. Downing, S. Reliability: on the reproducibility of assessment data. Med Educ. 2004;38(9):1006-1012.CrossRefGoogle ScholarPubMed
62. Haladyna, TM, Downing, SM. Construct‐irrelevant variance in high‐stakes testing. Educational Measurement: Issues and Practice. 2004;23(1):17-27.CrossRefGoogle Scholar
63. Streiner, D, Norman, G. Health Measurement Scales: A Practical Guide to their Development and Use, Fourth ed. New York, New York USA: Oxford University Press; 2008.CrossRefGoogle Scholar
64. Brennan, R. Performance assessments from the perspective of generalizability theory. Appl Psychol Meas. 2000;24(4):339-354.CrossRefGoogle Scholar
65. Eva, KW. On the generality of specificity. Med Educ. 2003;37(7):587-588.CrossRefGoogle ScholarPubMed
66. Eva, KW, Neville, AJ, Norman, GR. Exploring the etiology of content specificity: factors influencing analogic transfer and problem solving. Acad Med. 1998;73(10):S1-S5.CrossRefGoogle ScholarPubMed
67. Van der Vleuten, C, Swanson, D. Assessment of clinical skills with standardized patients: state of the art. Teach Learn Med. 1990;2(2):58-76.CrossRefGoogle Scholar
68. Newble, D, Swanson, D. Psychometric characteristics of the objective structured clinical examination. Med Educ. 1988;22(4):325-334.CrossRefGoogle ScholarPubMed
69. Van der Vleuten, CP. When I say… context specificity. Med Educ. 2014;48(3):234-235.CrossRefGoogle ScholarPubMed
70. Downing, S, Yudkowsky, R. Assessment in Health Professions Education. New York, New York USA: Taylor & Francis; 2009.CrossRefGoogle Scholar
71. Kane, MT. “Validity.” In: Educational Measurement. Brennan BL, (ed). Westport, Connecticut USA: Praeger Pub. Text; 2006.Google Scholar
72. Downing, S. Validity: on the meaningful interpretation of assessment data. Med Educ. 2003;37(9):830-837.CrossRefGoogle ScholarPubMed
73. Schuwirth, LW, Van der Vleuten, CP. Programmatic assessment and Kane’s validity perspective. Med Educ. 2012;46(1):38-48.CrossRefGoogle ScholarPubMed
74. Kane, M. Current concerns in validity theory. Journal of Educational Measurement. 2001;38(4):319-342.CrossRefGoogle Scholar
75. Kane, M. Validating score interpretations and uses. Language Testing. 2012;29(1):3-17.CrossRefGoogle Scholar
76. Kane, MT. An argument-based approach to validity. Psychol Bull. 1992;112(3):527-535.CrossRefGoogle Scholar
77. Downing, S, Haladyna, T. Validity threats: overcoming interference with proposed interpretations of assessment data. Med Educ. 2004;38(3):327-333.CrossRefGoogle ScholarPubMed
78. Downing, S. Threats to the validity of clinical teaching assessments: what about rater error? Med Educ. 2005;39(4):353-355.CrossRefGoogle Scholar
79. Haladyna, T, Downing, S. Construct irrelevant variance in high stakes testing. Educational Measurement: Issues and Practice. 2004;23(1):17-27.CrossRefGoogle Scholar
80. Hodges, B, Hanson, M, McNaughton, N, Reghr, G. Creating, monitoring, and improving a psychiatry OSCE: a guide for faculty. Acad Psychiatry. 2002;26(3):134-161.CrossRefGoogle ScholarPubMed
81. Harden, R, et al. Assessment of clinical competence using objective structured examination. Br Med J. 1975;1(5955):447-451.CrossRefGoogle ScholarPubMed
82. Harden, R. What is an OSCE? Med Teach. 1988;10(1):19-22.CrossRefGoogle ScholarPubMed
83. Turner, J, Dankoski, M. Objective structured clinical exams: a critical review. Fam Med. 2008;40(8):574-578.Google ScholarPubMed
84. Hatala, RM, Stevenson, M, Downie, WW, Wilson, GM. Modification of an OSCE format to enhance patient continuity in a high-stakes assessment of clinical performance. BMC Med Educ. 2011;11(1):23.CrossRefGoogle Scholar
85. Eva, KW, Rosenfeld, J, Reiter, HI, Norman, GR. An admissions OSCE: the multiple mini interview. Med Educ. 2004;38(3):314-326.CrossRefGoogle ScholarPubMed
86. Ryan, S, Stevenson, K, Hassell, AB. Assessment of clinical nurse specialists in rheumatology using an OSCE. Musculoskeletal Care. 2007;5(3):119-129.CrossRefGoogle ScholarPubMed
87. Rushforth, HE. Objective structured clinical examination (OSCE): review of literature and implications for nursing education. Nurse Educ Today. 2007;27(5):481-490.CrossRefGoogle ScholarPubMed
88. Park, RS, Chibnall, JT, Blaskiewicz, RJ, Furman, GE, Powell, JK, Mohr, CJ. Construct validity of an objective structured clinical examination (OSCE) in psychiatry: associations with the clinical skills examination and other indicators. Acad Psychiatry. 2004;28(2):122-128.CrossRefGoogle ScholarPubMed
89. Manogue, M, Brown, G. Developing and implementing an OSCE in dentistry. Eur J Dent Educ. 1998;2(2):51-57.CrossRefGoogle ScholarPubMed
90. Reznick, R, Blackmore, D, Dauphinee, WD, Rothman, AI, Smee, S. Large-scale high-stakes testing with an OSCE: report from the Medical Council of Canada. Acad Med. 1996;71(1):S19-S21.CrossRefGoogle ScholarPubMed
91. Smee, SM, Dauphinee, WD, Blackmore, DE, Rothman, AI, Reznick, RK, Des Marchais, J. A sequenced OSCE for licensure: administrative issues, results and myths. Adv Health Sci Edu Theory Pract. 2003;8(3):223-236.CrossRefGoogle ScholarPubMed
92. Lee, YS. OSCE for the medical licensing examination in Korea. The Kaohsiung J Med Sci. 2008;24(12):646-650.CrossRefGoogle ScholarPubMed
93. Boulet, JR, Smee, SM, Dillon, GF, Gimpel, JR. The use of standardized patient assessments for certification and licensure decisions. Simul Healthc. 2009;4(1):35-42.CrossRefGoogle ScholarPubMed
94. Patrício, MF, Juliao, M, Fareleira, F, Carniero, AV. Is the OSCE a feasible tool to assess competencies in undergraduate medical education? Med Teach. 2013;35(6):503-514.CrossRefGoogle ScholarPubMed
95. Sloan, DA, Donnelly, MB, Schwartz, RW, Strodel, WE. The objective structured clinical examination (OSCE). The new gold standard for evaluating postgraduate clinical performance. Ann Surg. 1995;222(6):735-742.CrossRefGoogle ScholarPubMed
96. Tamblyn, R, Abrahamowicz, M, Brailovsky, C, et al. Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA. 1998;280(11):989-996.CrossRefGoogle ScholarPubMed
97. Ruesseler, M, Weinlich, M, Byhahn, C, et al. Increased authenticity in practical assessment using emergency case OSCE stations. Adv Health Sci Edu Theory Pract. 2010;15(1):81-95.CrossRefGoogle ScholarPubMed
98. Vleuten, C, Norman, G, Graaff, E. Pitfalls in the pursuit of objectivity: issues of reliability. Med Educ. 1991;25(2):110-118.CrossRefGoogle ScholarPubMed
99. Norman, G, Vleuten, C, Graaff, E. Pitfalls in the pursuit of objectivity: issues of validity, efficiency, and acceptability. Med Educ. 1991;25(2):119-126.CrossRefGoogle ScholarPubMed
100. Regehr, G, MacRae, H, Reznick, RK, Szalay, D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med. 1998;73(9):993-997.CrossRefGoogle Scholar
101. Ilgen, JS, Ma, IW, Hatala, R, Cook, DA. A systematic review of validity evidence for checklists versus global rating scales in simulation‐based assessment. Med Educ. 2015;49(2):161-173.CrossRefGoogle ScholarPubMed
102. Hodges, B, Regehr, G, McNaughton, N, Tiberius, R, Hanson, M. OSCE checklists do not capture increasing levels of expertise. Acad Med. 1999;74(10):1129-1134.CrossRefGoogle Scholar
103. Malau-Aduli, BS, Mulcahy, S, Warnecke, E, et al. Inter-rater reliability: comparison of checklist and global scoring for OSCEs. Creat Educ. 2012;3(6A):937-942.CrossRefGoogle Scholar
104. Eva, KW, Hodges, BD. Scylla or Charybdis? Can we navigate between objectification and judgment in assessment? Med Educ. 2012;46(9):914-919.CrossRefGoogle ScholarPubMed
105. Hodges, B. Assessment in the post-psychometric era: learning to love the subjective and collective. Med Teach. 2013;35(7):564-568.CrossRefGoogle ScholarPubMed
106. Hanson, JL, Rosenberg, AA, Lane, JL. Narrative descriptions should replace grades and numerical ratings for clinical performance in medical education in the United States. Front Psychol. 2013;21(4):668.Google Scholar
107. Gingerich, A, Kogan, J, Yeates, P, Govaerts, M, Holmboe, E. Seeing the “black box” differently: assessor cognition from three research perspectives. Med Educ. 2014;48(11):1055-1068.CrossRefGoogle ScholarPubMed
108. Tavares, W, Eva, KW. Exploring the impact of mental workload on rater-based assessments. Adv Health Sci Edu Theory Pract. 2013;18(2):291-303.CrossRefGoogle ScholarPubMed
109. Norman, G. When I say...reliability. Med Educ. 2014;48(9):946-947.CrossRefGoogle Scholar
110. Ginsburg, S, Regehr, G, Lingard, L, Eva, KW. Reading between the lines: faculty interpretations of narrative evaluation comments. Med Educ. 2015;49(3):296-306.CrossRefGoogle ScholarPubMed
111. Wood, TJ. Exploring the role of first impressions in rater-based assessments. Adv Health Sci Edu Theory Pract. 2014;19(3):409-427.CrossRefGoogle ScholarPubMed
112. Gingerich, A, Regehr, G, Eva, KW. Rater-based assessments as social judgments: rethinking the etiology of rater errors. Acad Med. 2011;86(10):S1-S7.CrossRefGoogle ScholarPubMed
113. Govaerts, MJ, Van de Wiel, MW, Schuwirth, LW, Van der Vleuten, CP, Muijtjens, AM. Workplace-based assessment: raters’ performance theories and constructs. Adv Health Sci Edu Theory Pract. 2013;18(3):375-396.CrossRefGoogle ScholarPubMed
114. Yeates, P, O’Neill, P, Mann, K, Eva, K. Seeing the same thing differently. Adv Health Sci Edu Theory Pract. 2013;18(3):325-341.CrossRefGoogle ScholarPubMed
115. Ginsburg, S, McIlroy, J, Oulanova, O, Eva, K, Regehr, G. Toward authentic clinical evaluation: pitfalls in the pursuit of competency. Acad Med. 2010;85(5):780.CrossRefGoogle ScholarPubMed
116. Crossley, J, Johnson, G, Booth, J, Wade, W. Good questions, good answers: construct alignment improves the performance of workplace based assessment scales. Med Educ. 2011;45(6):560-569.CrossRefGoogle ScholarPubMed