Skip to main content Accessibility help
×
Home

Information:

  • Access
  • Cited by 2

Actions:

      • Send article to Kindle

        To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Self-awareness of computed tomography ordering in the emergency department
        Available formats
        ×

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Self-awareness of computed tomography ordering in the emergency department
        Available formats
        ×

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Self-awareness of computed tomography ordering in the emergency department
        Available formats
        ×
Export citation

Abstract

Objectives

Physician variation in the use of computed tomography (CT) is concerning due to the risks of ionizing radiation, cost, and downstream effects of unnecessary testing. The objectives of this study were to describe variation in CT-ordering rates among emergency physicians (EPs), to measure correlation between perceived and actual CT-ordering rates, to assess attitudes that influence decisions to order imaging tests, and to identify EP attitudes associated with higher CT utilization.

Methods

This study was a retrospective review of imaging and administrative billing records at two emergency department sites of a tertiary care adult teaching hospital. The study also included a cross-sectional survey of EPs at this hospital. We asked physicians about their perceived ordering behaviour, and what factors influenced their decision to order a CT. We examined correlations between perceived and actual CT-ordering rates. We adjusted ordering rates for shift distribution using a logistic regression model and identified outlier physicians whose ordering rate was significantly lower or higher than expected. We used multivariable regression analysis to determine which survey responses predicted higher CT utilization.

Results

During the study period, 59 EPs saw 45,854 patients, and ordered 6,609 CTs — a mean ordering rate of 14.4% (standard deviation (SD)=4.3%). The ordering rate for individual physicians ranged from 5.9% to 25.9%. Of the 59 EPs, 13 EPs were low-ordering outliers; 12 were high-ordering outliers. Forty-five EPs (76.3%) completed the survey. Mean perceived ordering rate was 12.6%, and was weakly correlated with actual ordering (r=0.19, p=0.21). 42 EPs (93.3%) believed they ordered “about the same” or “fewer” CTs than their peers. Of the 17 EPs in the two highest ordering quintiles, only 3 (18%) knew they were high orderers. In the multivariable analysis, higher ordering was associated with increasing strength of response to the following predictors: medico-legal risk (relative risk [RR]=1.18, 95% CI: 1.03–1.21), risk of contrast (RR=1.14, 95% CI: 1.07–1.22), what colleagues would do (RR=1.09, 95% CI: 0.99–1.19), risk of missing a diagnosis (RR=1.08, 95% CI: 0.98–1.21), and patient wishes (RR=1.07, 95% CI: 0.97–1.17).

Conclusions

There is large variation in CT ordering among EPs. Physicians’ self-reported ordering rate correlates poorly with actual ordering. High CT orderers were rarely aware that they ordered more than their colleagues. Higher rates of ordering were observed among physicians who reported increased concern with 1) risk of missing a diagnosis, 2) medico-legal risk, 3) risk of contrast, 4) patient wishes, and 5) what colleagues would do.

INTRODUCTION

Utilization of computed tomography (CT) scans in the emergency department (ED) has increased by more than 300% over the last two decades.Reference Kocher, Meurer, Fazel, Scott and Krumholz 1 Studies have demonstrated that emergency physicians (EPs) vary in their test-ordering behaviour.Reference Jain, Elon, Johnson, Frank and Deguzman 2 - Reference Marin, Weaver and Barnato 7 Inter-physician variation in the ordering of any test may introduce unnecessary costsReference Cutler, Skinner, Stern and Wennberg 8 or subject a patient to the risks of over-diagnosis, under-diagnosis, false positives, or further unnecessary testing.Reference Patz, Pinsky and Gatsonis 9 - Reference Thompson, Wojcik, Grant and Ko 11 Variation in the use of CT scans is of special significance because CT scans expose patients to ionizing radiation at doses that likely increase a patient’s lifetime risk of cancer.Reference Brenner and Hall 12 - Reference Eisenberg, Afilalo and Lawler 17 This practice variation may imply that some patients are being exposed to radiation unnecessarily.

Inter-physician variation has significant cost implications. Within a given department, test ordering may vary by two- to threefold, with corresponding variation in length of stay, admission rate, and downstream testing.Reference Klassen, Reed and Stiell 3 , Reference Levine, Moore, Franck, Li and Kuehl 4

While studies have demonstrated variation in CT ordering among EPs, these studies have not determined whether individual physicians are aware of their utilization rates.Reference Klassen, Reed and Stiell 3 - Reference Marin, Weaver and Barnato 7

Proposed drivers of physician variation include fear of malpractice and physician risk tolerance.Reference Katz, Williams and Brown 18 - Reference Wong, Kowalenko and Roahen-Harrison 24 However, physician variation is not completely understood.Reference Rohacek, Buatsi and Szucs-Farkas 23 , Reference Feinglass, Martin and Sen 25 The objectives of this study were to describe variation in CT-ordering rates among EPs, to measure the correlation between perceived and actual CT-ordering rates, to assess attitudes that might influence a decision to order an imaging test, and to identify which of these attitudes are associated with higher CT utilization.

METHODS

Study design

This study included a retrospective review of imaging and administrative billing records at two ED sites of a tertiary-care adult teaching hospital (Ottawa, Ontario), and a cross-sectional survey of EPs at this hospital. We obtained approval from The Ottawa Hospital Research Ethics Board.

Study setting and population

The Ottawa Hospital includes two urban EDs with a combined 140,000 annual patient visits and is a referral centre for the region of Eastern Ontario. One site is the regional trauma and stroke centre; the other site is the regional oncology centre. The hospital is the principal teaching site for an emergency medicine residency. Medical students and junior residents from other specialties rotate through the ED. The study population included all patients attending the ED during the study period, as well as all the EPs employed at this institution during that time.

Study protocol

Departmental billing records were used to determine the number of patients seen per EP for the months of January, March, June, and September of 2012. These months were selected due to the availability of detailed billing data. An imaging database, the Picture Archiving and Communication System (PACS), was employed to determine the number of CTs ordered by each EP. To order a CT, EPs or residents completed paper requisitions that were scanned into PACS. We searched PACS by ordering physician and then read the scanned requisition to minimize errors in order attribution. If no attending physician was listed on the requisition, the CT was excluded from our analysis.

Although most residents at our institution discuss CT ordering decisions with attending EPs, senior emergency medicine residents in their 3rd, 4th, or 5th year might not. CTs ordered after 11:00 pm were often delayed until morning, but were still attributed to the original ordering physician.

We defined a single CT study by the scan protocol at our institution. For instance, a CT of the abdomen and pelvis was considered a single study. If a patient had more than one CT ordered with different protocols (e.g., CT head and CT pulmonary angiogram), we counted both CTs.

We did not review the ED charts or collect patient data (e.g., ESI/CTAS score, patient age, diagnosis).

The survey was conducted in three stages. First, study authors chose, by consensus, factors that might plausibly influence a physician’s decision to order a CT. Second, the survey was pilot-tested with senior emergency medicine residents to ensure comprehensibility, and to obtain feedback on redundant questions, which we eliminated. Third, the paper survey was delivered to the hospital mailbox of all EPs at The Ottawa Hospital in June 2013. The survey had 21 questions and took about 5 minutes to complete. Two reminder emails, to encourage completion, were sent monthly after initial survey distribution. Physicians who completed the survey were entered in a draw for a $50 gift certificate. The final survey included: 1) demographic data; 2) 10 self-reported factors that influence physicians’ ordering practices rated on a 4-point Likert-type scale; 3) five beliefs on CT utilization rated on a 5-point Likert-type scale; and 4) an estimate of the perceived rate of self-ordering.

Data analysis

Ordering rates were calculated for each physician and expressed as a cumulative percentage over the study period. The average across physicians was calculated as an unweighted mean. The ED duty schedule at The Ottawa Hospital includes four different shift types: urgent (ambulatory care); observation (stretchers without cardiac monitors); resuscitation (trauma, resuscitation, and cardiac monitoring); and night (entire department). We adjusted each physician’s ordering rate to account for differences in shift distribution, and to identify the “outlier” status of physicians, using the approach described by DeLong et al.Reference DeLong, Peterson and DeLong 26 We constructed a random-effects logistic regression model in which shift type was a fixed covariate, and physicians were considered random effects, using SAS software (v. 9.1). The physician random effects were estimated as best linear unbiased predictors, which are shrunken estimates, closer to the mean CT-ordering rate, and are more robust against small sample sizes. The random-effects estimates obtained from the logistical regression model were expressed as observed versus expected ratios of CT-ordering rate, with their 95% confidence intervals, for each physician. Physicians whose confidence intervals did not include 1 were considered outliers.

The correlation between perceived and actual CT-ordering rates was calculated using Pearson’s correlation coefficient, as well as using mean difference with a 95% confidence interval (CI). Physicians were categorized into quintiles based on their actual ordering rates and compared to the perceived rates based on survey responses.

We used multivariable regression to identify important independent predictors of higher CT utilization.Reference Harrell 27 Physicians who had seen fewer than 300 patients during the study period were excluded from the multivariable model. Potential predictors of higher CT utilization were first assessed using bivariate tests of association; in particular, demographic characteristics (physician sex, categorized age) were assessed using chi-squared tests. Ten survey factors measured on a Likert-type scale were analyzed as ordinal variables, in which 0=not influential and 3=heavily influential, and assessed using chi-squared tests/Cochran–Armitage trend tests. Shift type was analyzed as a categorical variable and assessed using chi-squared tests. Multicollinearity among candidate predictor variables was assessed using a variance-clustering algorithm,Reference Nelson 28 and variables involved in near-linear dependencies were excluded from consideration. Variables significantly associated (p<0.05) with CT utilization were considered for entry into a multivariable generalized linear regression model using the log-link function and Poisson distribution. The CT-ordering rate for each shift type was entered as the dependent variable, and the natural log of the number of patients seen was specified as an offset term. We accounted for over-dispersion by using a Pearson scale parameter. Stepwise backward elimination with an alpha level for staying in the model set at 0.2 was used to determine independent predictors of CT utilization. Regression coefficients were estimated as relative risks with CIs. All analyses were completed using Statistical Analysis Software (v. 9.3; SAS Institute, Cary, NC).

RESULTS

Physician characteristics

CT ordering data were available for 59 physicians, of whom 45 (76%) completed the survey. Another six physicians completed the survey, but CT ordering data were unavailable for them (due to recent entry to practice or leave from clinical duties). Demographic data were therefore available for 51 EPs, which are summarized in Table 1.

Table 1 Characteristics of physicians included in the study (n=51)

CCFP (EM)=Canadian College of Family Physicians–Emergency Medicine certificate (three-year training program); FRCPC=Fellow of the Royal College of Physicians of Canada (five-year training program).

Chart review findings and actual ordering rates

Unadjusted CT-ordering rates are displayed in Figure 1. During the study period, 59 EPs saw 45,854 patients and ordered 6,609 CTs—an ordering rate of 14.4% (standard deviation (SD)=4.3%). The average numbers of patients and CTs per physician were 777 and 112, respectively. The ordering rate for individual physicians ranged from 5.9 to 25.9%. There were 11 physicians whose CT-ordering rate was >1 SD (or 3 physicians whose CT-ordering rate was >2 SD) above the mean.

Figure 1 Variation in CT-ordering rates among emergency physicians.

The ratios of observed to expected CT-ordering rates, adjusted for shift type, are presented in Figure 2, with their CIs. MDs are numbered in the same order as in Figure 1. In this analysis, 13 physicians were low-ordering outliers, and 12 physicians were high-ordering outliers. Three physicians whose unadjusted rate was in the lowest quintile and two physicians whose unadjusted rate was in the highest quintile were not outliers in this analysis. This likely reflects uneven shift allotment within the department. Seven physicians who saw fewer than 300 patients during the study period were excluded from the multivariable model, but none were outliers.

Figure 2 Standardized CT-ordering rate with 95% confidence limits.

Survey findings

CT data were available for 59 of 65 EPs. Of those, 45 completed the survey, for an overall survey response rate of 76.3%. The mean perceived ordering rate was 12.6% (SD=7.1). As anticipated, the perceived ordering rate was weakly correlated with actual ordering for individual physicians (r=0.19, CI: –0.11 to 0.46, p=0.21) (Figure 3).

Figure 3 Scatter plot of actual versus perceived CT-ordering rates.

The distributions of physicians, categorized into quintiles of perceived and actual CT ordering, are depicted in Figure 4.

Figure 4 Perceived versus actual ordering rates, by quintile, from lowest (Q1) to highest (Q5).

Only 29% of EPs were able to predict their ordering behaviour to the correct quintile.

From 45 EPs, 42 responders (93.3%) believed they ordered “about the same” or “fewer” CTs than their peers. Of the 17 EPs in the two highest-ordering quintiles, only 3 (18%) knew they were high CT utilizers (Figure 4).

The self-reported factors that influenced physicians’ decisions to order CTs are presented in Figure 5.

Figure 5 Self-reported factors that influence physicians’ decision to order CT.

Emergency physicians’ views on CT ordering are presented in Figure 6. A total of 15 (29%) of respondents disagreed that they should order fewer CTs, but only 5 (10%) disagreed with the statement that colleagues should order fewer CTs. A total of 9 (18%) agreed that they should order fewer CTs, whereas 17 (33%) agreed that colleagues should order fewer CTs. The majority of physicians agreed that CT increases the risk of cancer (90%) and that EPs order more CTs than they used to (96%).

Figure 6 Physicians’ views on CT ordering.

Multivariable regression analysis

The following variables were not significantly associated with CT-ordering rates using bivariate tests of association: physician gender (p=0.42), age (p=0.19), residency program (p=0.20), and residency location (p=0.69). These variables were therefore excluded from the multivariable model. Moreover, the Likert-type scale responses to “lifetime number of CTs a patient already had” and “risk of radiation” were collinear, as were “ED flow” and “medico-legal risk.” We therefore excluded “lifetime number of CTs a patient already had” and “ED flow” from consideration in the multivariable model. Higher Likert-type scale responses to the following predictors were associated with higher ordering rates: medico-legal risk (RR=1.18, CI: 1.03–1.21); risk of contrast (RR=1.14, CI: 1.07–1.22); what colleagues would do (RR=1.09, CI: 0.99–1.19); risk of missing a diagnosis (RR=1.08, CI: 0.98–1.21); and patient wishes (RR=1.07, CI: 0.97–1.17).

DISCUSSION

We demonstrate important variation in CT-ordering rates and that a subset of physicians order far more CT scans than their colleagues. This is consistent with the existing literature.Reference Levine, Moore, Franck, Li and Kuehl 4 , Reference Prevedello, Raja and Zane 6 , Reference Kim, Sidney, Klingman and Johnston 29

This study also examined physicians’ self-awareness of their test-ordering behaviour. To our knowledge, this is the first ED-based study to address this question. When physicians were asked to estimate their absolute rate of CT ordering (as a percentage), we found only a weak correlation with their actual rate of ordering. This suggests that physicians have a poor awareness of their personal ordering rate.

Although physicians might not be able to estimate their absolute ordering rate, we considered that they might know how they compare to peers. When asked to rank their ordering rate compared to colleagues, the majority of physicians believed that their practice was in the middle quintile. Only 3 of 45 physicians believed they fell in the upper two quintiles.

Despite this apparent lack of self-awareness, physicians were aware of the risks of radiation. Nonetheless, only 18% felt that they should order fewer CT scans. In comparison, 33% of physicians felt that their colleagues should order fewer CT scans. These data suggest that few physicians in this study recognized over-utilization in their own practice, but rather attributed it to others. Physicians clearly believe that the tests they order are appropriate. We were unable to assess the appropriateness of the CT scans ordered in the study, as we did not collect clinical information on the patients. Previous studies, however, suggest that positive findings are no more common in high-ordering centres,Reference Klassen, Reed and Stiell 3 , Reference Stiell, Wells and Vandemheen 30 so that some CT scans are likely inappropriate.

The problem of variation in physician practice remains vexing. Numerous studies and several reviews have evaluated strategies to improve physician test-ordering behaviour.Reference Oxman, Thomson, Davis and Haynes 31 , Reference Solomon, Hashimoto, Daltroy and Liang 32 Approaches to modifying physician behaviour include the use of clinical practice guidelines, academic detailing, audit and feedback, pay-for-performance, and computer-based decision support, but no strategy has demonstrated consistent success in all settings.Reference Hartig and Allison 33 Our findings suggest that the highest CT orderers did not realize that they ordered more than their colleagues. Although peer comparisons have not been shown to be useful for changing practiceReference Jamtvedt, Young, Kristoffersen, O’Brien and Oxman 34 in other ways, high-ordering physicians might be more willing to reduce the use of CT, as it has the potential to directly harm patients.

EDs should also consider their aggregate utilization compared to comparator hospitals. Although different hospitals serve different populations, it is unlikely that patient factors can account for all institutional variability,Reference Klassen, Reed and Stiell 3 , Reference Mannix, Meehan, Monuteaux and Bachur 5 , Reference Marin, Weaver and Barnato 7 , Reference Cutler, Skinner, Stern and Wennberg 8 , Reference Stiell, Wells and Vandemheen 30 and there is room for entire departments to change their practice.

In the absence of clear evidence-based approaches to the evaluation of many chief complaints, some inter-physician variation in practice is to be expected. This study, however, demonstrates that measuring raw ordering rates alone may not be sufficient to identify outlier physicians, as some EPs in our cohort with apparently high or low ordering rates were not outliers once we adjusted for shift distribution. Adjusting for individual patient factors might also be considered, but this was beyond the scope of our study. Nonetheless, our study, like others, suggests that there is a subset of high-ordering physicians who might benefit from measures to reduce utilization. There is also a subset of low-ordering physicians. There is some evidence to suggest that low orderers do not miss important diagnoses,Reference Stiell, Wells and Vandemheen 30 but more research is necessary.

We attempted to find predictors of high CT ordering using muiltivariate regression analysis. We did not find that physician age, sex, and training program were associated with higher CT ordering. However, some responses on our survey of physician attitudes predicted higher rates of CT ordering. In our reduced multivariate model, higher CT ordering was predicted by a stronger response to questions rating concern for medical-legal risk, risk of contrast, what colleagues would do, patient wishes, or risk of missing a diagnosis. These findings are not all intuitive. In Canada, medico-legal risk is comparatively low compared to that in the United States,Reference Fleeter 35 but medico-legal concerns were still our strongest predictor of increased imaging. Concern regarding risk of contrast should predict decreased CT ordering, but, paradoxically, we found the opposite.

Factors that have previously been shown to be associated with test utilization include the patient’s geographic location,Reference Klassen, Reed and Stiell 3 , Reference Mannix, Meehan, Monuteaux and Bachur 5 , Reference Cutler, Skinner, Stern and Wennberg 8 , Reference Stiell, Wells and Vandemheen 30 , Reference Couchman, Forjuoh, Reis, Bartels and Lindzey 36 race,Reference Pezzin, Keyl and Green 37 , Reference Harris 38 insurance status,Reference Couchman, Forjuoh, Reis, Bartels and Lindzey 36 and physician involvement in guideline development.Reference Verstappen, ter Riet and Dubois 39 Physician risk tolerance and fear of malpractice have also been associated with differences in test utilization.Reference Katz, Williams and Brown 18 , Reference Pines, Hollander and Isserman 19 , Reference Cheng, Wu, Syue, Yen and Wu 22 - Reference Wong, Kowalenko and Roahen-Harrison 24 A study of internal medicine physiciansReference Feinglass, Martin and Sen 25 found that variation in practice style accounted for a 40% variation in hospital charges.

There is unlikely to be a single solution to the problem of physician variation in test utilization.Reference Oxman, Thomson, Davis and Haynes 31 This study demonstrates that physicians have poor awareness of how their test ordering compares to that of their colleagues. We argue that it is unlikely that physicians with high levels of utilization will be willing to change test-ordering behaviour without feedback. Improved physician self-awareness could be part of the solution.

LIMITATIONS

This study is limited by data collection from two EDs within a single Canadian university system. Our results may not be generalizable to other hospitals or other countries. Although we collected data on over 6,500 CTs, the average number of CTs per physician was 112.

Some CTs might have been ordered by residents without prior approval by their attending physicians. Nevertheless, we believe that the effect on CT-ordering rate would have been small. Most CT-ordering decisions are discussed with attending physicians. Given the size of our department and residency program, attending physicians work with senior residents on a minority of shifts. Residents are not preferentially assigned to any physician, so the effect of senior resident ordering should be evenly distributed.

The survey included responses from only 45 physicians; a larger number of physicians might have produced a stronger multivariate model. Physicians who did not respond to the survey might have held different attitudes toward CT ordering from those who responded, and this could have affected the results of our multivariable regression. In our model, risk of contrast was associated with higher, not lower, ordering rates. This may represent a limitation of the model, as risk of contrast should be a deterrent to CT ordering.

Finally, the factors that physicians rated on our survey instrument were chosen by consensus among the investigators. It is possible that other factors we did not measure would be more predictive. It is also possible that physician self-report of the importance of a factor is an inaccurate representation of their behaviour.

CONCLUSIONS

There is large variation in CT ordering among EPs—in our centre, we measured a fourfold variation between the lowest and highest utilizers. Physicians’ self-reported ordering rates correlate poorly with actual ordering. Only a small proportion of high CT orderers were aware that they ordered more than their colleagues. Higher rates of ordering were observed among physicians who reported increased concern with: 1) risk of missing a diagnosis, 2) medico-legal risk, 3) risk of contrast, 4) patient wishes, and 5) what colleagues would do.

Acknowledgements

Aspects of this research study were presented in abstract form at the Meeting of the Society of Academic Emergency Medicine (SAEM), Dallas, Texas, May 2014 and the Meeting of the Canadian Association of Emergency Physicians (CAEP), Ottawa, Ontario, June 2014.

Competing interests: This study was funded through a research grant from the Department of Emergency Medicine, University of Ottawa, and through a student research grant from the Faculty of Medicine, University of Ottawa. The authors have no financial relationships that may represent a conflict of interest.

REFERENCES

1. Kocher, KE, Meurer, WJ, Fazel, R, Scott, PA, Krumholz, HM, et al. National trends in use of computed tomography in the emergency department. Ann Emerg Med 2011;58(5):452-462.
2. Jain, S, Elon, LK, Johnson, BA, Frank, G, Deguzman, M. Physician practice variation in the pediatric emergency department and its impact on resource use and quality of care. Pediatr Emerg Care 2010;26(12):902-908.
3. Klassen, TP, Reed, MH, Stiell, IG, et al. Variation in utilization of computed tomography scanning for the investigation of minor head trauma in children: a Canadian experience. Acad Emerg Med 2000;7(7):739-744.
4. Levine, MB, Moore, AB, Franck, C, Li, J, Kuehl, DR. Variation in use of all types of computed tomography by emergency physicians. Am J Emerg Med 2013;31(10):1437-1442.
5. Mannix, R, Meehan, WP, Monuteaux, MC, Bachur, RG. Computed tomography for minor head injury: variation and trends in major United States pediatric emergency departments. J Pediatr 2012;160(1):136-139.
6. Prevedello, LM, Raja, AS, Zane, RD, et al. Variation in use of head computed tomography by emergency physicians. Am J Med 2012;125(4):356-364.
7. Marin, JR, Weaver, MD, Barnato, AE, et al. Variation in emergency department head computed tomography use for pediatric head trauma. Acad Emerg Med 2014;21(9):987-995.
8. Cutler, D, Skinner, J, Stern, AD, Wennberg, D. Physician beliefs and patient preferences: a new look at regional variation in health care spending. National Bureau of Economic Research Working Paper No. 19320; 2013. Available at: http://www.nber.org/papers/w19320.pdf (accessed June 3, 2017).
9. Patz, EF, Pinsky, P, Gatsonis, C, et al. Overdiagnosis in low-dose computed tomography screening for lung cancer. JAMA Intern Med 2014;174(2):269-274.
10. Lumbreras, B, Donat, L, Hernández-Aguado, I. Incidental findings in imaging diagnostic tests: a systematic review. Br J Radiol 2010;83(988):276-289.
11. Thompson, RJ, Wojcik, SM, Grant, WD, Ko, PY. Incidental findings on CT scans in the emergency department. Emerg Med Int 2011;2011:624847. 10.1155/2011/624847.
12. Brenner, DJ, Hall, EJ. Computed tomography—an increasing source of radiation exposure. N Engl J Med 2007;357(222):2277-2284.
13. Cardis, E, Vrijheid, M, Blettner, M, et al. Risk of cancer after low doses of ionising radiation: retrospective cohort study in 15 countries. BMJ 2005;331(7508):77.
14. Berrington de Gonzalez, A, Mahesh, M, Kim, KP, et al. Projected cancer risks from computed tomographic scans performed in the United States in 2007. Arch Intern Med 2009;169(22):2071-2077.
15. Mathews, JD, Forsythe, AV, Brady, Z, et al. Cancer risk in 680,000 people exposed to computed tomography scans in childhood or adolescence: data linkage study of 11 million Australians. BMJ 2013;346:f2360.
16. Rajaraman, P, Simpson, J, Neta, G, et al. Early life exposure to diagnostic radiation and ultrasound scans and risk of childhood cancer: case-control study. BMJ 2011;342:d472.
17. Eisenberg, MJ, Afilalo, J, Lawler, PR, et al. Cancer risk related to low-dose ionizing radiation from cardiac imaging in patients after acute myocardial infarction. CMAJ 2011;183(4):430-436.
18. Katz, DA, Williams, GC, Brown, RL, et al. Emergency physicians’ fear of malpractice in evaluating patients with possible acute cardiac ischemia. Ann Emerg Med 2005;46(6):525-533.
19. Pines, JM, Hollander, JE, Isserman, JA, et al. The association between physician risk tolerance and imaging use in abdominal pain. Am J Emerg Med 2009;27(5):552-557.
20. Pines, JM, Isserman, JA, Szyld, D, et al. The effect of physician risk tolerance and the presence of an observation unit on decision making for ED patients with chest pain. Am J Emerg Med 2010;28(7):771-779.
21. Smith-Bindman, R, McCulloch, CE, Ding, A, Quale, C, Chu, PW. Diagnostic imaging rates for head injury in the ED and states’ medical malpractice tort reforms. Am J Emerg Med 2011;29(6):656-664.
22. Cheng, FJ, Wu, CH, Syue, YJ, Yen, PC, Wu, KH. Association of physician risk tolerance with ED CT use for isolated dizziness/vertigo patients. Am J Emerg Med 2014;32(10):1284-1288.
23. Rohacek, M, Buatsi, J, Szucs-Farkas, Z, et al. Ordering CT pulmonary angiography to exclude pulmonary embolism: defense versus evidence in the emergency room. Intensive Care Med 2012;38(8):1345-1351.
24. Wong, AC, Kowalenko, T, Roahen-Harrison, S, et al. A survey of emergency physicians’ fear of malpractice and its association with the decision to order computed tomography scans for children with minor head trauma. Pediatr Emerg Care 2011;27(3):182-185.
25. Feinglass, J, Martin, GJ, Sen, A. The financial effect of physician practice style on hospital resource use. Health Serv Res 1991;26(2):183-205.
26. DeLong, ER, Peterson, ED, DeLong, DM, et al. Comparing risk-adjustment methods for provider profiling. Stat Med 1997;16(23):2645-2664.
27. Harrell, FE. Regression Modelling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. New York: Springer Science and Business Media; 2001.
28. Nelson, B. Variable reduction for modelling using PROC VARCLUS. Paper presented at the SUGI 26 conference. Available at: http://www2.sas.com/proceedings/sugi26/p261-26.pdf (accessed March 30, 2016).
29. Kim, AS, Sidney, S, Klingman, JG, Johnston, SC. Practice variation in neuroimaging to evaluate dizziness in the ED. Am J Emerg Med 2012;30(5):665-672.
30. Stiell, IG, Wells, GA, Vandemheen, K, et al. Variation in ED use of computed tomography for patients with minor head injury. Ann Emerg Med 1997;30(1):14-22.
31. Oxman, AD, Thomson, MA, Davis, DA, Haynes, RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. CMAJ 1995;153(10):1423-1431.
32. Solomon, DH, Hashimoto, H, Daltroy, L, Liang, MH. Techniques to improve physicians’ use of diagnostic tests: a new conceptual framework. JAMA 1998;280(23):2020-2027.
33. Hartig, JR, Allison, J. Physician performance improvement: an overview of methodologies. Clin Exp Rheumatol 2007;25(6 Suppl 47):50-54.
34. Jamtvedt, G, Young, JM, Kristoffersen, DT, O’Brien, MA, Oxman, AD. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2006;(2):CD000259.
35. Fleeter, TG. Worldwide trends in medical liability. Paper presented at AAOS Now, September 2011. Available at: http://www.aaos.org/news/aaosnow/sep11/managing4.asp (accessed June 4, 2017).
36. Couchman, GR, Forjuoh, SN, Reis, MD, Bartels, G, Lindzey, D. Variation in MRI/CT utilization among family physicians and general internists in a multi-specialty group practice. Med Sci Monit 2005;11(3):MT19-MT25.
37. Pezzin, LE, Keyl, PM, Green, GB. Disparities in the emergency department evaluation of chest pain patients. Acad Emerg Med 2007;14(2):149-156.
38. Harris, MI. Racial and ethnic differences in health care access and health outcomes for adults with type 2 diabetes. Diabetes Care 2001;24(3):454-459.
39. Verstappen, WH, ter Riet, G, Dubois, WI, et al. Variation in test order in test ordering behaviour of GPs: professional or context-related factors? Fam Pract 2004;21(4):387-395.