Skip to main content Accessibility help
×
Home

Do regulated resident working hours affect medical graduate education? Trends in the American psychiatry board pass rates pre- and post-2003 duty hours regulations

  • Gaurav Jain (a1), Kristina Dzara (a2), Mir Nadeem Mazhar (a3) and Manisha Punwani (a4)

Abstract

Aims and method

To assess trends of the American Board of Psychiatry and Neurology examination pass rates before and after the 2003 duty hours regulations (DHR). We obtained the pass rates for part I and II for years 2000–2010. Data were divided pre-DHR (2000–2003) and post-DHR (2007–2010).

Results

During the pre-DHR period, first- and multiple-attempt group pass rates were 80.7% and 39.0% which changed in the post-DHR period to 89.7% and 39.1% respectively. Similarly for the part II exam, the pre-DHR first- and multiple-attempt group pass rates were 60.2% and 43.5% respectively, which increased to 78.7% and 53.8%, among the post-DHR group. Overall, there was a significant increase in the first-attempt candidates pass rates for parts I and II, whereas multiple-attempt candidates did not benefit as strongly.

Clinical implications

The results suggest that the 2003 DHR may have had a positive impact on examination-based medical knowledge in psychiatry.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Do regulated resident working hours affect medical graduate education? Trends in the American psychiatry board pass rates pre- and post-2003 duty hours regulations
      Available formats
      ×

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Do regulated resident working hours affect medical graduate education? Trends in the American psychiatry board pass rates pre- and post-2003 duty hours regulations
      Available formats
      ×

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Do regulated resident working hours affect medical graduate education? Trends in the American psychiatry board pass rates pre- and post-2003 duty hours regulations
      Available formats
      ×

Copyright

This is an Open Access article, distributed under the terms of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.

Corresponding author

Mir Nadeem Mazhar (mnm1@queensu.ca)

Footnotes

Hide All

The abstract of this study was presented as a poster at the 2012 American Psychiatric Association Annual Meeting in Philadelphia, USA. The views expressed in this paper do not reflect the official policy or position of the American Board of Psychiatry and Neurology.

Declaration of interests

None.

Footnotes

References

Hide All
1 Philibert, I, Friedmann, P, Williams, WT. New requirements for resident duty hours. JAMA 2002; 288: 1112–4.
2 Nasca, TJ, Day, SH, Amis, ES. The new recommendations on duty hours from the ACGME Task Force. N Engl J Med 2010; 363: e3.
3 Steinbrook, R. The debate over residents' work hours. N Engl J Med 2002; 347: 1296–302.
4 Charap, M. Reducing resident work hours: unproven assumptions and unforeseen outcomes. Ann Intern Med 2004; 140: 814–5.
5 West, CP, Cook, RJ, Popkave, C, Kolars, JC. Perceived impact of duty hours regulations: a survey of residents and program directors. Am J Med 2007; 120: 644–8.
6 Fletcher, KE, Reed, DA, Arora, VM. Patient safety, resident education and resident well-being following implementation of the 2003 ACGME duty hour rules. J Gen Intern Med 2011; 26: 907–19.
7 Jamal, MH, Rousseau, MC, Hanna, WC, Doi, SA, Meterissian, S, Snell, L. Effect of the ACGME duty hours restrictions on surgical residents and faculty: a systematic review. Acad Med 2011; 86: 3442.
8 Moonesinghe, SR, Lowery, J, Shahi, N, Millen, A, Beard, JD. Impact of reduction in working hours for doctors in training on postgraduate medical education and patients' outcomes: systematic review. BMJ 2011; 342: d1580.
9 Skeff, KM, Ezeji-Okoye, S, Pompei, P, Rockson, S. Benefits of resident work hours regulation. Ann Intern Med 2004; 140: 816–7.
10 Fletcher, KE, Underwood III, W, Davis, SQ, Mangrulkar, RS, McMahon, LF, Saint, S. Effects of work hour reduction on residents' lives: a systematic review. JAMA 2005; 294: 1088–100.
11 Jagsi, R, Shapiro, J, Weissman, JS, Dorer, DJ, Weinstein, DF. The educational impact of ACGME limits on resident and fellow duty hours: a pre-post survey study. Acad Med 2006; 81: 1059–68.
12 Myers, JS, Bellini, LM, Morris, JB, Graham, D, Katz, J, Potts, JR, et al Internal medicine and general surgery residents' attitudes about the ACGME duty hours regulations: a multicenter study. Acad Med 2006; 81: 1052–8.
13 Sutton, B. The need to define and study resident duty hours. Acad Psychiatr 2004; 28: 253–5.
14 Rasminsky, S, Lomonaco, A, Auchincloss, E. Work hours regulations for house staff in psychiatry: bad or good for residency training? Acad Psychiatr 2008; 32: 5460.
15 Sattar, SP, Basith, F, Madison, J, Bhatia, SC. New ACGME work-hour guidelines and their impact on current residency training practices. Acad Psychiatr 2005; 29: 279–82.
16 Drolet, BC, Spalluto, LB,Fischer, SA. Residents' perspectives on ACGME regulation of supervision and duty hours – a national survey. N Engl J Med 2010; 363: e34.
17 American Board of Psychiatry and Neurology. 2010 Information for Applicants: Initial Certification – Psychiatry. ABPN, 2009. Available at http://www.abpn.com/downloads/ifas/IFA_Cert_Psych_2010.pdf (accessed 28 Nov 2011).
18 American Board of Medical Specialties. 2010 ABMS Certificate Statistics: pp. 1011. ABMS, 2010.
19 Sneider, EB, Larkin, AC, Shah, SA. Has the 80-hour workweek improved surgical resident education in New England? J Surg Educ 2009; 66: 140–5.
20 Faulkner, LR. A Selective History of the ABPN: It s' Déjà Vu All Over Again (presentation). ABPN, 2009. Available at http://www.abpn.com/downloads/presentations/Selective%20History%20ABPN%20092109%20[Compatibility%20Mode].pdf (accessed 28 Nov 2011).
21 Batalden, P, Leach, D, Swing, S, Dreyfus, H, Dreyfus, S. General competencies and accreditation in graduate medical education. Health Affairs 2002; 21: 103–11.

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed

Do regulated resident working hours affect medical graduate education? Trends in the American psychiatry board pass rates pre- and post-2003 duty hours regulations

  • Gaurav Jain (a1), Kristina Dzara (a2), Mir Nadeem Mazhar (a3) and Manisha Punwani (a4)
Submit a response

eLetters

Reply to letter submitted by Dr Osman

Mir N Mazhar, Psychiatrist, Queen's University, Canada
Gaurav Jain, Fellow, University of Pittsburgh, USA
Kristina Dzara, Research Associate, Yale University School of Medicine, USA.
08 July 2015

We wish to thank Dr Osman for the critical review of our publication (1) and the opportunity to clarify a number of points from our work. Although we agree that Dr Osman accurately describes an overall trend of improvement over time (from 2000 through 2010), we disagree that the statistical analysis we originally preformed was flawed. Applying time series analysis to our study is inappropriate. Dr Osman is proposing a completely separate study by including data from 2004 – 2006 in his analysis and utilizing the aggregated values. Dr Osman used data altered from that of the scope of our methodology (1) and is thus comparing different things. We purposefully chose not to include data from 2004 – 2006 in our analysis, as we were interested in how test-takers performed prior to and after duty hours’ regulation (DHR) were fully implemented. We agree with Dr Osman that if the DHR encouraged an upward trend in pass rates, the improvement over time in the 2004 - 2007 group was expected. However, Dr Osman changed the research question by eliminating the effect of improvement over time. We endeavoured to test for improvement over time, whereas Dr Osman wishes to remove it through the use of time series models by smoothing the data and eliminating the distinct changes between the two time cohorts in exchange for a moving average of the data. Dr Osman’s point that there is an upwards trend in the 2007-2010 cohorts may strengthen the argument that the DHR improved results, but was not the focus of our study. We suggest that in order to test Dr Osman’s hypothesis, then possibly the linear models of the three distinct timeframes (2000-2003, 2004-2007, 2008-2011) would need to be tested. However, this was also not done in the time series analysis.

Importantly, Dr Osman’s response regarding the need for time-series analysis, negative binomial regression, or autoregressive integrated moving average (ARIMA) models highlight the methodological conflict faced by researchers, where easily grasped robust methodologies are rejected in favor of more complicated, hard to interpret models. Clearly, we noted that ‘because of data limitations, we were unable to compute full statistical models controlling for all confounding variables which may have had an effect on pass rates’ and made no claim to be the final authority on this topic. We are confident that our methodology, analysis, and findings are true to the data, and provide insight into the effect of the 2003 DHR on examination-based medical knowledge in Psychiatry. We are honoured to contribute to the 2003 DHR conversation and encourage other researchers to continue assessing potential impacts of DHR on resident education.

Acknowledgement: the authors wish to sincerely thank Dr Dianne Groll, Ph.D, Assistant Professor, Department of Psychiatry, Queen’s University, Canada, for her insightful statistical guidance.

Reference:

1. Jain G, Dzara K, Mazhar MN, Punwani M. Do regulated resident working hours affect medical graduate education? Trends in the American psychiatry board pass rates pre- and post-2003 duty hours regulations. Psychiatric Bulletin 2014 Dec;38(6):299-302.
... More

Conflict of interest: None Declared

Write a reply

RE: Accounting for upward trends in the American psychiatry board pass rates pre- and post-2003 duty hours regulations by using the Monte Carlo exact test.

We congratulate Jain and colleagues (1) on their interesting article investigating the effect of duty hours’ regulation (DHR) on the academic performance of psychiatric trainees. Plenty of research has evaluated the concerns raised by medical educators of the potential negative effect DHR may have on clinical aspects of professionalism and patient safety (2). Jain and colleagues’ results indicated an overall improvement in terms of pass rates for part I and part II examinations organised annually by the American Board of Psychiatry and Neurology.

However, the statistical analysis was flawed and did not fully capture the apparent upward trend in pass rates between 2000 and 2010. Such trends imply autocorrelation between successive annual pass rates. This can be visualized by plotting an autocorrelation function, which indicated an autocorrelation of 0.695 at lag 1 (p value= 0.03). A rigorous interrupted time series analysis could have accounted for such trends (3) better than the simple calculation of an odds ratio undertaken in the paper. Such rich time series data carry plenty of information that can be analysed effectively using designs such as negative binomial regression or Auto-Regressive Integrated Moving Average ARIMA models available on all statistical software packages.

Another method, to account for the positive autocorrelation is to apply a generalized linear model to the pass rates for the 2000-2003 period and apply another generalized linear model to the pass rates for the 2007-2010 period. Hence, using the 2000-2003 model coefficients as the null hypothesis, we can investigate whether the 2007-2010 model coefficients are significantly different.

Examining the data pertaining to pass rates between 2000 and 2003 uncovers a statistically significant linear relationship of the form odds of Pass Rate=e^(-340.41071+0.17037×Year). However, upon fitting a generalized regression model to the 2007-2010 data, we get a different relationship of odds of Pass Rate=e^(-132.13977 +0.06256×Year). This was not statistically different from the 2000-2007 in terms of both the slope and the level of the trend (p value =0.40).

Another, more advanced technique, is the Monte Carlo exact test. This test is useful when the sample size is not large enough, as is the case in comparing the very short time series 2000-2003 and 2007-2010, for satisfactory asymptotic approximations (4). A recent paper used the Monte Carlo exact test to estimate p values for improvement of quality indicators over a three year interval (5). We take the significant linear relationship of the form odds of Pass Rate=e^(-340.41071+0.17037×Year) that governs the behaviour of the 2000-2003 data as our null hypothesis. Should such relationship hold on the 2007-2010 interval, would we get an odds ratio quite different from the observed one? This is the essence of this method. We simulate a large number (a thousand) of odds ratios under this null hypothesis assumption. This enables us to assign a p value to the odds ratio we have observed, accounting for the linear relationship that was overlooked by the authors.

Moreover, the authors have rounded up the pass rates and then re-used it to calculate odds ratios. For instance, we calculated the odds ratio dividing the more accurate crude pass numbers by the total number of attempters. We found it to be 1.845 rather than the reported 1.865! Nonetheless, taking 1.865 as an observed odds ratio, we calculated a Monte Carlo p value of 0.09, a statistically insignificant result. This is clearly different from the p <0.0001 reported in the paper. Similarly, for the first time attempters the p value we simulated was 0.39, very insignificant. We found a different p value because of our effort in taking the upward trend into account before investigating any evidence of improvement in academic performance. Thus, the conclusion reached by the authors that DHR has led to an overall improvement of pass rates for part I and part II examinations is left open to question.

References:

1.Jain G, Dzara K, Mazhar MN, Punwani M. Do regulated resident working hours affect medical graduate education? Trends in the American psychiatry board pass rates pre- and post-2003 duty hours regulations. Psychiatric Bulletin (2014). 2014 Dec;38(6):299-302.

2.Szymczak JE1, Brooks JV, Volpp KG, Bosk CL. To leave or to lie? Are concerns about a shift-work mentality and eroding professionalism as a result of duty-hour rules justified? The Milbank Quaterly 2010 Sep;88(3):350-81.

3.Penfold RB, Zhang F. Use of interrupted time series analysis in evaluating health care quality improvements. Academic Pediatrics. 2013 Nov-Dec;13(6 Suppl):S38-44.

4.Yuan A, Bonney GE. Exact test of Hardy-Weinberg equilibrium by Markov chain Monte Carlo. Mathematical Medicine and Biology. 2003 Dec;20(4):327-40.

5.Laronga C, Gray JE, Siegel EM, Lee JH, Fulp WJ, Fletcher M, Schreiber F, Brown R, Levine R, Cartwright T, Abesada-Terk G Jr, Kim G, Alemany C, Faig D, Sharp P, Markham MJ, Shibata D, Malafa M, Jacobsen PB. Florida Initiative for Quality Cancer Care: improvements in breast cancer quality indicators during a 3-year interval. Journal of the American College of Surgeons. 2014 Oct;219(4):638-45.e1

... More

Conflict of interest: None Declared

Write a reply

×

Reply to: Submit a response


Your details


Conflicting interests

Do you have any conflicting interests? *