Skip to main content Accessibility help
×
Home

Information:

  • Access

Actions:

      • Send article to Kindle

        To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Reformulations of practice: beyond experience in paramedic airway management
        Available formats
        ×

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Reformulations of practice: beyond experience in paramedic airway management
        Available formats
        ×

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Reformulations of practice: beyond experience in paramedic airway management
        Available formats
        ×
Export citation

Abstract

Objective

“Deliberate practice” and “feedback” are necessary for the development of expertise. We explored clinical performance in settings where these features are inconsistent or limited, hypothesizing that even in algorithmic domains of practice, clinical performance reformulates in ways that may threaten patient safety, and that experience fails to predict performance.

Methods

Paramedics participated in two recorded simulation sessions involving airway management, which were analyzed three ways: first, we identified variations in “decision paths” by coding the actions of the participants according to an airway management algorithm. Second, we identified cognitive schemas driving behavior using qualitative descriptive analysis. Third, clinical performances were evaluated using a global rating scale, checklist, and time to achieve ventilation; the relationship between experience and these metrics was assessed using Pearson’s correlation.

Results

Thirty participants completed a total of 59 simulations. Mean experience was 7.2 (SD=5.8) years. We observed highly variable practice patterns and identified idiosyncratic decision paths and schemas governing practice. We revealed problematic performance deficiencies related to situation awareness, decision making, and procedural skills. There was no association between experience and clinical performance (Scenario 1: r=0.13, p=0.47; Scenario 2: r=−0.10, p=0.58), or the number of errors (Scenario 1: r=.10, p=0.57; Scenario 2: r=0.25, p=0.17) or the time to achieve ventilation (Scenario 1: r=0.53, p=0.78; Scenario 2: r=0.27, p=0.15).

Conclusion

Clinical performance was highly variable when approaching an algorithmic problem, and procedural and cognitive errors were not attenuated by provider experience. These findings suggest reformulations of practice emerge in settings where feedback and deliberate practice are limited.

INTRODUCTION

Deliberate practice is widely acknowledged as important for the development of expertise 1 and requires that learners engage in effortful activities to incrementally close gaps between current and optimal skill performance while supported by feedback from expert mentors 2 , 3 , suggesting experience alone is insufficient in developing expertise. In education settings these supportive features are readily available; however, after entry to practice, these features may become inconsistent or limited 4 - 6 .

Paramedics practice in settings where important elements in the development of expertise may be limited. For instance, there is a high degree of independence, access to external feedback is limited, expert mentors and senior clinicians are unavailable and opportunities to engage in deliberate practice are infrequent. Paramedics are left to self-assess, a process which may be inherently limited 7 , reflecting on their actions with the risk of mistakenly re-enforcing poor technical or cognitive strategies in the absence of clear errors. This creates situations where practice variations may emerge that satisfy rather than optimize performance.

We chose to examine this unique phenomenon—practicing in settings where features supportive of the development of expertise are limited—using pre-hospital airway management as an example. Out-of-hospital airway management by paramedics is associated with low success rates 8 - 10 , high rates of complications 11 - 15 and, in some cases, poor patient outcomes 16 , 17 . Some have suggested that entry-to-practice training (e.g., manikin v. cadaver v. live patient) 18 - 21 and/or skill degradation 22 - 25 might explain these findings. However, to our knowledge, this issue of practicing in settings where features that support the development of expertise are unavailable has not been considered within the context of this problem.

Our primary research question was therefore: Among a group of paramedics with similar entry-to-practice training, what happens to clinical performance in settings where access to features that support the development of expertise are inconsistent or limited? As a secondary research question, we also asked: What is the relationship between experience and clinical performance among our sample? Based on our conceptual framework, we predicted that the paramedics’ clinical performances would vary, and we further hypothesized that experience would not be a predictor of clinical acumen.

METHODS

Overview

We used a convergent mixed methods study design, involving both quantitative and qualitative data collection and analysis. 26 We invited paramedics with varying degrees of experience, both in years of service and place of employment, to complete two full clinical cases (involving airway management) in simulation.

This study was conducted at the Centennial College Inter-professional Simulation Center, located in Toronto, Ontario, Canada and was approved by the Centennial College Research Ethics Board (REB #184).

Participant pool and context of practice

In Ontario, Canada, two levels of paramedics provide pre-hospital emergency care: primary and advanced care paramedics. Primary Care Paramedics (PCPs) complete a two-year community college diploma program and provide basic life support (e.g., cardiopulmonary resuscitation (CPR), intravenous access, “symptom relief” medications). Advanced Care Paramedics (ACPs) complete a third year of community college training and provide advanced life support, including oral and naso-tracheal intubation, supraglottic airway insertion and surgical cricothryroidotomy. In Ontario, ACP training programs may vary in structure but are consistent in accreditation requirements 27 with respect to content. Specific to airway management, ACP students learn in progressively complex environments including simulation-based, clinical (e.g., operating room) and field settings, with standardized minimum competency thresholds for each setting. Upon completion of their training, ACP students are required to complete a provincial written examination for certification and undergo additional knowledge and performance-based testing through their respective Regional Base Hospital (regulatory bodies responsible for paramedic practice and education; n=7) for authorization to practice. Once practicing, ACPs complete 24-hours of classroom and simulation-based mandatory continuing education per year. In Ontario, all paramedics practice under a set of provincially standardized advanced and basic life support patient care standards thus ensuring a consistent scope of practice. 28 , 29

Participant recruitment and selection

We used convenience sampling to recruit our participants. 30 , 31 Eligibility requirements included: (a) practicing ACPs; and (b) trained at an accredited college in Ontario. We excluded paramedics who have cross-trained in another health profession and also paramedics at the critical care level where context of training and care is different with respect to airway management. We distributed invitations to participate in the study via workplace email. All participants completed a questionnaire that included basic demographic information (age, sex, years of experience), annual exposure to airway management (intubation, supraglottic airway insertion, cricothyroidotomy) and whether the participant had completed any specialized airway management courses. Our intent was to recruit a sample of participants who, as much as reasonably possible, were similar except for years of clinical experience.

Simulation cases

We created two scenarios for this study, designed for their ability to approximate both a very complicated (Scenario 1) and routine (Scenario 2) airway management case. The order of case presentation was randomized. Both scenarios involved adult patients (using Laerdal SimMan®) requiring advanced airway management (e.g., supraglottic airway insertion or endotracheal intubation), however in Scenario 1, advanced airway management was scripted to be unsuccessful (i.e., a “can’t intubate”, “can’t ventilate” crisis). Both cases were pilot tested by experienced paramedics with teaching responsibilities at the advanced care level to ensure content validity. See Appendix 1 for additional case details.

Procedures

We asked the participants to work through the cases as they would in real clinical practice. Each case included two standardized actors that played the role of PCPs to assist the participants as needed. Both cases were scripted to continue until effective ventilation was achieved (a surgical airway for Scenario 1 or successful advanced airway placement for Scenario 2) or 15 minutes had elapsed, whichever occurred first. All performances were recorded using four strategically positioned cameras.

Outcome measures and data analysis

Based on our conceptual framework, we decided a priori to explore three outcome variables: (1) variation in the decision paths (i.e., sequence of steps); (2) schemas (i.e., cognitive frameworks that help organize and process information) that inform how the cases are managed; and (3) the relationship between experience (in years) and clinical performance (see Figure 1).

Figure 1 Study Process Overview.

Decision paths

We defined decision paths as the sequence of steps taken in managing the case and the alignment (or lack thereof) with a criterion standard, in this case, an airway management algorithm published by Wang et al. 32 (see Figure 2). To our knowledge, this is the only published algorithm specifically intended to guide out-of-hospital airway management by paramedics. This algorithm is not taught during ACP training and is not used in clinical practice in our context and was intended to be a ‘neutral’ reference point from which to assess between-participant variation in management decisions. We modified the algorithm slightly by substituting the term “advanced airway” for “endotracheal intubation” since supraglottic airway devices are considered equivalent in our setting.

Figure 2 Algorithm for Prehospital Airway Management by Wang et al, 2005.

Two investigators (JM, SD) used video review to code and sequence the participants’ management decisions based on observable behaviors. This required both researchers to agree on each coded behavior with a third researcher (WT) providing a final decision if consensus could not be reached. We then mapped a series of “ideal” pathways through the scenarios using the algorithm and then qualitatively analyzed the coded sequences of participant actions by comparing them with the ideal pathways and with each other for concordance and emerging commonalities, using ordered letters to indicate participant paths. At times, participants performed actions that were not part of the algorithm; these were coded as “X” interventions (Table 1).

Table 1 Legend for coded behaviors

Schema-based behaviors

To explore this cognitive component of practice, each video was reviewed for observable behaviors that could suggest schemas that the participants held and applied when managing the scenarios. Two investigators (JM, SD) independently recorded hand written memos while observing the scenarios, intending to capture the granularity of the participants’ clinical performance. These notes were analyzed using qualitative descriptive analysis 33 , 34 , a process that involves organizing observations of events into themes, with JM and SD meeting regularly during the coding process and resolving discrepancies through consensus.

Experience as a Predictor

Finally, our conceptual framework also suggests that experience alone would not predict expertise. To explore this relationship, we compared experience practicing at the advanced care level with three markers of clinical performance: (a) a previously validated paramedic-specific global rating scale (GRS) 35 , 36 , (b) a task-specific checklist and (c) time to achieve effective ventilation (defined above). The GRS evaluates clinical performance across seven dimensions (situation awareness, history gathering, patient assessment, decision making, resource utilization, communication and procedural skills) considered representative of paramedic practice using a 7-point adjectival scale (i.e., 1=“unsafe”; 7=“exceptional”) 36 (see Appendix 2). The task-specific checklist was constructed using accepted definitions for procedural errors in airway management drawn from the literature (Table 2). Finally, time from patient contact until effective ventilation was measured in minutes and seconds.

Table 2 Procedural error definitions

Two investigators (JM, SD) with experience using the GRS evaluated each performance using both the global rating scale and the checklist. For the first three participants, the investigators evaluated the performances jointly to achieve consensus on performance expectations and scoring; all subsequent evaluations took place independently. Inter-rater reliability was calculated using the inter-class coefficient for both the GRS scores (by dimension) and the incidence of errors (using the checklist). The relationship between participant experience (in years) and clinical performance was assessed using correlation analyses between (a) each dimension on the GRS; (b) the number of errors committed by each participant; and (c) and the time to achieve ventilation. Assuming a conservative correlation coefficient of 0.45, we required approximately 27 participants to obtain 95% confidence intervals of 0.1. We hypothesized that experience would fail to correlate with any clinical performance variable.

RESULTS

Thirty ACPs from six paramedic services in southern Ontario, representing a mix of urban, suburban and rural contexts with large variation in call volumes completed a total of 59 simulations. One participant declined to complete the second scenario for personal reasons. Twenty-five (83%) participants were male; the participants had an average of 12.0 (SD=6.6) years of experience as paramedics and an average of 7.2 (SD 5.8) years experience practicing at the advanced care level. Full demographic characteristics are presented in Table 3.

Table 3 Participant demographics

SGA = Supraglottic Airway.

Scenario content validity

To check the content validity of our scenarios, we conducted a number of analyses to assess the difficulty of the scenarios and whether the scenarios were perceived to be realistic by the participants. We report these results in Appendix 3; in brief, Scenario 1 was more challenging than scenario 2 (as expected) and the scenarios were felt to convey a high degree of physical, conceptual and emotional realism.

Decision paths

The participants’ coded behaviors are presented in Tables 4 (scenario 1) and 5 (scenario 2); we achieved complete agreement on the coded behaviors between the researchers with no irreconcilable discrepancies. Only one participant (during Scenario 2) followed one of the “ideal” pathways exactly. The participants varied extensively in the sequencing of steps, interventions chosen and points at which the scenarios were (or were not) successfully resolved. In contrast to what should have been a linear and algorithmic problem to solve, we observed that no two participants approached the clinical problem in the same way.

Table 4 Coded actions for Scenario 1

Table 5 Coded action for Scenario 2

Schema-based behaviors

Our analysis revealed a number of potential schemas (underlying cognitive processes that appeared to be driving behavior) that were applied by participants. We broadly categorized these as relating to situation awareness, decision making and procedural sills.

Related to situation awareness, we observed that participants had a tendency to focus in on specific tasks, often at the expense of an awareness of the underlying primary problem—a failed airway resulting in critical hypoxemia. For example, we observed that participants would attend to the patient’s deteriorating vital signs (e.g., bradycardia and hypotension), not realizing that these were symptoms of the airway problem and as a result, would fail to establish a patent airway. This also resulted in patient deterioration going undetected.

Regarding decision-making, we also observed schemas related to decision or care thresholds and strategies. For example, we observed variability related to when advanced airway use was considered or attempted, this was especially true of the threshold at which the participants would consider a surgical airway—a decision point usually determined a priori during training. Also, for many we observed a reversal of airway management strategies with participants often attempting more advanced procedures (e.g., intubation) prior to preparatory or less invasive techniques (e.g., use of adjuncts or airway positioning), despite standards of practice recommending the opposite.

Finally, we observed problematic schemas designed to mitigate challenges with procedural skills. For instance, we observed participants failing to follow best practice guidelines for intubating technique (poor patient positioning, levering the laryngoscope on the patient’s teeth, inappropriate equipment selection, etc.). An illustrative example involved a number of participants who would knowingly leave an esophageal endotracheal tube in place and then attempt to insert a second endotracheal tube (sometimes with a laryngoscope, sometimes without), in the (mistaken) belief it would enter the trachea.

In summary, participants seem to adopt individualized schemas over time that appear to develop through their clinical practice that in many cases may only satisfy (and in some ways threaten) rather than optimize patient care and safety.

Experience as a predictor

Table 6 lists the average GRS scores, number of errors committed per participant (based on a checklist) and average time to achieve ventilation for both scenarios. We achieved excellent inter-observer agreement in the global rating scale evaluation for the domains considered particularly important for the study (ICC=0.89–0.96 for situation awareness, decision making and procedural skills) and for the number of errors observed per participant (ICC=0.89–0.93). We found no correlation between years of experience at the advanced care level and overall GRS scores (Scenario 1: r=0.13, p=0.47; Scenario 2: r= −0.10, p=0.58), or the average number of errors committed (Scenario 1: r=0.10, p=0.57; Scenario 2: r=0.25, p=0.17) or finally, the time to achieve effective ventilation (Scenario 1: r=0.53, p=0.78; Scenario 2: r=0.27, p=0.15). We repeated the correlation analyses using Spearman’s correlation and obtained similar results.

Table 6 Clinical performance analysis results

ICC = inter-class correlation, SD = standard deviation. SA = situation awareness; HG = history gathering; PA = patient assessment; DM = decision making; RU = resource utilization; CM = communication; PS = procedural skill; OV = overall score.

* All GRS scores evaluated using 7-point adjectival scale; 1=“unsafe”, 7=“exceptional”.

DISCUSSION

Paramedicine allows for practice and the accumulation of experience, both of which are necessary for professional development, but lacks other essential features (i.e., access to feedback, deliberate practice, and proximal supervision) for the development of expertise. Our goal in this study was to explore how clinical experience in scenarios where these features are limited or inconsistent affects performance. Our results suggest that even when faced with a highly algorithmic problem, variations or reformulations of practice can emerge. While previous work has identified variation between clinicians 37 , 38 , particularly with respect to guideline adherence 39 , to our knowledge the concept of reformulations of practice has not been described. A search of the education literature revealed no previous use of the term reformulations of practice. We propose defining reformulations of practice as behaviors, schemas, and decision paths that evolve over time through experience but unpredictably and often detrimentally. While some variation between providers would be expected to occur naturally, in an algorithmic domain of practice like airway management, the amount of variation we observed was significant. Further, experience failed to mitigate the effects, collectively suggesting that inherent profession-level structures may be placing a ceiling on the development of paramedic expertise. Our study has important implications for the profession regarding the maintenance of competence.

While there is still debate within the education community regarding the optimal timing and amount of feedback 40 - 43 , the concept that feedback enhances learning has been well established 44 . Feedback can be intrinsic (i.e., immediate, haptic) or external (i.e., provided by an expert mentor or coach) and is intended to identify deficiencies in performance and prescribe corrective guidance. 41 , 45 , 46 That we observed such heterogeneity in performance among a largely homogenous group of clinicians suggests that feedback is still occurring but that it is intrinsically—and internally—derived. This was evidenced in a highly algorithmic domain, where presumably reformulations of practice are less likely. This raises concerns for less technical or more cognitively dependent skills that are susceptible to the same limitations. The contextual issues in paramedicine may contribute challenges in maintaining competence, especially with complex but infrequently used skills. This requires careful consideration to the structuring of practice and learning opportunities available to paramedics.

Deliberate practice has widely been acknowledged as a necessary condition for learning and expertise. 47 , 48 It has also been suggested that achieving a level of proficiency beyond what is initially required for competence (i.e., overlearning) may help to slow the degradation of skills. 4 , 5 , 49 Another option includes promoting and supporting a culture of ongoing, individualized simulation-based learning. This provides a useful platform to maintain low frequency, high complexity skills by allowing for deliberate practice in a realistic environment while neutralizing concerns over patient safety. 50 - 53 Finally, another strategy may involve recording clinical encounters and using the footage to facilitate educational debriefings after high acuity cases. This strategy has been used successfully in neonatal 54 and trauma resuscitations 55 , 56 ; however, some logistical issues would need to be resolved for use in paramedicine. Implementing systems to support the ongoing competence of paramedics and other health care providers is a worthy goal for educators and administrators alike.

For clinicians and the research community, the results of our study might shed new light on paramedic airway management. Lack of experience, limited initial education, and low frequency of use have been previously suggested as problematic. 20 - 23 We suggest that even if those issues are resolved, until features associated with the development of expertise are addressed, problems may continue to persist.

LIMITATIONS

We acknowledge some limitations in our study. First, we assume that our participants were homogenous as a result of similar entry-to-practice training, however, without the benefit of a more structured longitudinal study, the possibility of confounding exists. Second, simulation can only ever serve as a surrogate for reality; it is possible that the behavior of the participants was influenced by the simulated nature of the study. Third, our sample size was limited. This raises the possibility of a type 2 error; however, this applies only to a secondary question conducted primarily as a test of our conceptual framework. Finally, the schemas governing performance were inferred—other schemas may be responsible for the behavior we observed.

CONCLUSION

Among a group of paramedics with similar entry-to-practice and ongoing training, we observed significant variation between providers and a number of potentially problematic reformulations of practice. We believe these variations may emerge as a function of limited opportunities for feedback and deliberate practice and suggest that administrators and educators consider the implications of this study in supporting the ongoing competence of clinicians performing complex skills.

Acknowledgements: JM and WT were responsible for the conception, design and implementation of the study, JM, SD, and MMcC were primarily responsible for analyzing the data, CL, AW, and EF provided important logistical support during data collection. All authors contributed to the development of the manuscript and approve its content. The authors wish to thank Peel Regional Paramedic Services, York Region Paramedic Services and Centennial College for important material contributions and also wish to express their gratitude to the dozens of paramedic student volunteers who assisted with the implementation of this study.

Competing Interests: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper. This study was funded by the Ontario Association of Paramedic Chiefs.

SUPPLEMENTARY MATERIAL

To view supplementary material for this article, please visit http://dx.doi.org/10.1017/cem.2016.371

REFERENCES

1. Ericsson, KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med 2003;79(10):S70-S81.
2. Ericsson, KA. Deliberate practice and acquisition of expert performance: a general overview. Acad Emerg Med 2008;15(11):988-994.
3. Ericsson, KA, Krampe, RT, Tesch-Romer, C. The role of deliberate practice in the acquisition of expert performance. Psychological Review 1993;100(3):363-406.
4. Pusic, MV, Kessler, D, Szyld, D, et al. Experience curves as an organizing framework for deliberate practice in emergency medicine learning. Acad Emerg Med 2012;19(12):1476-1480.
5. Pusic, MV, Pecaric, M, Boutis, K. How much practice is enough? Using learning curves to assess the deliberate practice of radiograph interpretation. Acad Med 2011;86(6):731-736.
6. Pusic, MV, Boutis, K, Hatala, R, et al. Learning Curves in Health Professions Education. Acad Med 2015;90(8):1034-1042.
7. Eva, K, Regehr, G. Self-assessment in the health professions: a reformulation and research agenda. Acad Med 2005;80(10):S46-S54.
8. Diggs, LA, Yusuf, JE, De Leo, G. An update on out-of-hospital airway management practices in the United States. Resuscitation 2014;85(7):885-892.
9. Wang, HE, Yealy, DM. How many attempts are required to accomplish out-of-hospital endotracheal intubation? Acad Emerg Med 2006;13(4):372-377.
10. Hubble, MW, Brown, L, Wilfong, DA, et al. A meta-analysis of prehospital airway control techniques part I: orotracheal and nasotracheal intubation success rates. Prehosp Emerg Care 2010;14(3):377-401.
11. Wang, HE, Lave, JR, Sirio, CA, et al. Paramedic intubation errors: isolated events or symptoms of larger problems? Health Aff 2006;25(2):501-509.
12. Lossius, HM, Roislien, J, Lockey, DJ. Patient safety in pre-hospital emergency tracheal intubation: a comprehensive meta-analysis of the intubation success rates of EMS providers. Crit Care 2012;16(1):R24.
13. Bowles, TM, Freshwater-Turner, DA, Janssen, DJ, et al. Out-of-theatre tracheal intubation: prospective multicentre study of clinical practice and adverse events. Br J Anaesth 2011;107(5):687-692.
14. Dunford, JV, Davis, DP, Ochs, M, et al. Incidence of transient hypoxia and pulse rate reactivity during paramedic rapid sequence intubation. Ann Emerg Med 200;42(6):721-728.
15. Wang, HE, Kupas, DF, Paris, PM, et al. Multivariate predictors of failed prehospital endotracheal intubation. Acad Emerg Med 2003;10(7):717-724.
16. Wang, HE, Sweeney, TA, O’Connor, RE, et al. Failed prehospital intubations: an analysis of emergency department courses and outcomes. Prehosp Emerg Care 2001;5(2):134-141.
17. von Goedecke, A, Herff, H, Paal, P, et al. Field airway management disasters. Anesth Analg 2007;104(3):481-483.
18. Walker, M, Jensen, JL, Leroux, Y, et al. The impact of intense airway management training on paramedic knowledge and confidence measured before, immediately after and at 6 and 12 months after training. Emerg Med J 2013;30(4):334-338.
19. Johnston, BD, Seitz, SR, Wang, HE. Limited opportunities for paramedic student endotracheal intubation training in the operating room. Acad Emerg Med 2006;13(10):1051-1055.
20. Wang, HE, Seitz, SR, Hostler, D, et al. Defining the learning curve for paramedic student endotracheal intubation. Prehosp Emerg Care 2005;9(2):156-162.
21. Westgard, BC, Peterson, BK, Salzman, JG, et al. Longitudinal and regional trends in paramedic student exposure to advanced airway placement: 2001-2011. Prehosp Emerg Care 2013;17(3):379-385.
22. Wang, HE, Balasubramani, GK, Cook, LJ, et al. Out-of-hospital endotracheal intubation experience and patient outcomes. Ann Emerg Med 2010;55(6):527-537 e6.
23. Wang, HE, Kupas, DF, Hostler, D, et al. Procedural experience with out-of-hospital endotracheal intubation. Crit Care Med 2005;33(8):1718-1721.
24. Pouliot, RC. Failed prehospital tracheal intubation: a matter of skill dilution? Anesth Analg 2010;110(5):1507-1508; author reply 1509.
25. Barnes, TA. Tracheal intubation: training and retention. Respir Care 2001;46(3):227-228.
26. Creswell, JW. Mixed Methods Approaches, In Research Design: Qualitative, Quantitative And Mixed Methods Approaches , (ed. Knight V). Oaks, CA: SAGE Publications Thousand; 2014.
27. Canadian Medical Association. Official List of Accredited Paramedic Program. 2016. Available at https://www.cma.ca/En/Pages/paramedicine.aspx (accessed January 25, 2015).
28. Paramedic Association of Canada. National Occupational Competency Profile; 2001. Available at: http://www.paramedic.ca/uploaded/web/documents/2011-10-31-Approved-NOCP-English-Master.pdf.
29. Emergency Health Services Branch, Ministry of Health and Long-Term Care. Advanced life support patient care standards: Version 3.2. Toronto: Emergency Health Services Branch, Ministry of Health and Long-Term Care; 2016.
30. Marshall, MN. Sampling for qualitative research. Fam Pract 1996;13(6):522-525.
31. Coyne, IT. Sampling in qualitative research. Purposeful and theoretical sampling; merging or clear boundaries? J Adv Nurs 1997;26:623-630.
32. Wang, HE, Kupas, DF, Greenwood, MJ, et al. An algorithmic approach to prehospital airway management. Prehosp Emerg Care 2005;9(2):145-155.
33. Sandelowski, M. Whatever happened to qualitative description? Res Nurs Health 2000;23:334-340.
34. Sandelowski, M. What’s in a name? Qualitative description revisited. Res Nurs Health 2010;33(1):77-84.
35. Tavares, W, Leblanc, VR, Mausz, J, et al. Simulation-based assessment of paramedics and performance in real clinical contexts. Prehosp Emerg Care 2014;18(1):116-122.
36. Tavares, W, Boet, S, Theriault, R, et al. Global rating scale for the assessment of paramedic clinical competence. Prehosp Emerg Care 2013;17(1):57-67.
37. Mercuri, M, Natarajan, MK, Norman, G, et al. An even smaller area variation: differing practice patterns among interventional cardiologists within a single high volume tertiary cardiac centre. Health Policy 2012;104(2):179-185.
38. Mercuri, M, Gafni, A. Medical practice variations: what the literature tells us (or does not) about what are warranted and unwarranted variations. J Eval Clin Pract 2011;17(4):671-677.
39. Mercuri, M, Sherbino, J, Sedran, RJ, et al. When guidelines don’t guide: the effect of patient context on management decisions based on clinical practice guidelines. Acad Med 2015;90(2):191-196.
40. Walsh, CM, Ling, SC, Wang, CS, et al. Concurrent versus terminal feedback: it may be better to wait. Acad Med 2009;84(10 Suppl):S54-S57.
41. Sigrist, R, Rauter, G, Riener, R, et al. Augmented visual, auditory, haptic, and multimodal feedback in motor learning: a review. Psychon Bull Rev 2013;20(1):21-53.
42. Hatala, R, Cook, DA, Zendejas, B, et al. Feedback for simulation-based procedural skills training: a meta-analysis and critical narrative synthesis. Adv Health Sci Educ Theory Pract 2014;19(2):251-272.
43. Cheng, A, Eppich, W, Grant, V, et al. Debriefing for technology-enhanced simulation: a systematic review and meta-analysis. Med Educ 2014;48(7):657-666.
44. Lefroy, J, Watling, C, Teunissen, PW, et al. Guidelines: the do’s, don’ts and don’t knows of feedback for clinical education. Perspect Med Educ 2015;4(6):284-299.
45. Ramani, S, Krackov, SK. Twelve tips for giving feedback effectively in the clinical environment. Med Teach 2012;34(10):787-791.
46. Wulf, G, Shea, C, Lewthwaite, R. Motor skill learning and performance: a review of influential factors. Med Educ 2010;44(1):75-84.
47. van de Wiel, MW, Van den Bossche, P, Janssen, S, et al. Exploring deliberate practice in medicine: how do physicians learn in the workplace? Adv Health Sci Educ Theory Pract 2011;16(1):81-95.
48. McGaghie, WC, Issenberg, SB, Cohen, ER, et al. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med 2011;86(6):706-711.
49. Pusic, MV, Brydges, R, Kessler, D, et al. What’s your best time? Chronometry in the learning of medical procedures. Med Educ 2014;48(5):479-488.
50. Cook, DA, Hatala, R, Brydges, R, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA 2011;306(9):978-998.
51. McGaghie, WC, Issenberg, SB, Petrusa, ER, et al. A critical review of simulation-based medical education research: 2003-2009. Med Educ 2010;44(1):50-63.
52. Harder, BN. Use of simulation in teaching and learning in health sciences: a systematic review. J Nurs Educ 2010;49(1):23-28.
53. McGaghie, WC, Issenberg, SB, Petrusa, ER., et al. A critical review of simulation-based mastery learning with translational outcomes. Med Educ 2014;48(4):375-385.
54. Carbine, DN, Finer, NN, Knodel, E, et al. Video recording as a means of evaluating neonatal resuscitation performance. Pediatrics 2000;106(4):654-658.
55. Scherer, LA, Chang, MC, Meredith, JW, et al. Videotape review leads to rapid and sustained learning. Am J Surg 2003;185(6):516-520.
56. Oakley, E, Stocker, S, Staubli, G, et al. Using video recording to identify management errors in pediatric trauma resuscitation. Pediatrics 2006;117(3):658-664.