Skip to main content Accessibility help


  • Craig R. Ramsay (a1), Lloyd Matowe (a2), Roberto Grilli (a3), Jeremy M. Grimshaw (a4) and Ruth E. Thomas (a1)...


Objectives: In an interrupted time series (ITS) design, data are collected at multiple instances over time before and after an intervention to detect whether the intervention has an effect significantly greater than the underlying secular trend. We critically reviewed the methodological quality of ITS designs using studies included in two systematic reviews (a review of mass media interventions and a review of guideline dissemination and implementation strategies).

Methods: Quality criteria were developed, and data were abstracted from each study. If the primary study analyzed the ITS design inappropriately, we reanalyzed the results by using time series regression.

Results: Twenty mass media studies and thirty-eight guideline studies were included. A total of 66% of ITS studies did not rule out the threat that another event could have occurred at the point of intervention. Thirty-three studies were reanalyzed, of which eight had significant preintervention trends. All of the studies were considered “effective” in the original report, but approximately half of the reanalyzed studies showed no statistically significant differences.

Conclusions: We demonstrated that ITS designs are often analyzed inappropriately, underpowered, and poorly reported in implementation research. We have illustrated a framework for appraising ITS designs, and more widespread adoption of this framework would strengthen reviews that use ITS designs.



Hide All
Avorn J, Soumerai SB, Taylor W et al. 1988 Reduction of incorrect antibiotic dosing through a structured educational order form. Arch Intern Med. 148: 1720- 1724.
Bareford D, Hayling A. 1990 Inappropriate use of laboratory services: Long term combined approach to modify request patterns. BMJ. 301: 1305- 1307.
Barnett GO, Winickoff R, Dorsey JL, Morgan MM, Lurie RS. 1978 Quality assurance through automated monitoring and concurrent feedback usinga computer-based medical information system. Med Care. 16: 962- 970.
Berbatis CG, Maher MJ, Plumridge RJ, Stoelwinder JU, Zubrick SR. 1982 Impact of a drug bulletin on prescribing oral analgesics in a teaching hospital. Am J Hosp Pharm. 39: 98- 100.
Bero L, Freemantle N, Grilli R et al. 1997 Checklist for data abstraction in EPOC reviews. Aberdeen: EPOC(mimeo)
Bero L, Grilli R, Grimshaw JM et al. 1998 Closing the gap between research and practice: An overview of systematic reviews of interventions to promote implementation findings by health care professionals. BMJ. 317: 465- 468.
Bero L, Grilli R, Grimshaw JM et al. 2002 The Cochrane Effective Practice and Organisation of care Group (EPOC) Module. In: The Cochrane Library. Oxford: Update Software Issue 1.
Blohm M, Herlitz J, Hartford M et al. 1992 Consequences of a media campaign focusing on delay in acute myocardial infarction. Am J Cardiol. 69: 411- 413.
Bonerandi JJ, Grob JJ, Cnudde N, Enel P, Gouvernet J. 1992 Campaign of early detection of melanoma in the Provence-Alpes-Cote-d'Azur area 1989. Lessons of an experience. Ann Dermatol Venereol. 119: 105- 109.
Box EP, Jenkins GM. 1976 Time series analysis. London: Holden-Day
Brasca A, Pigliacampo J, Pollastri E et al. 1987 An awareness campaign on the diagnosis and prevention of colon and rectal cancer. Planning and evaluation of its results. Acta Gastroenterol Latinoam. 17: 325- 329.
Brook RH, Williams KN. 1976 Effect of medical care review on the use of injections: A study of the New Mexico Experimental Medical Care Review Organization. Ann Intern Med. 85: 509- 515.
Brooks RJ. 1996 Implementing guidelines for eye care of diabetic patients: Results from an HMO intervention study. Am J Managed Care. 2: 365- 369.
Brufsky JW, Ross-Degnan D, Calabrese D, Gao X, Soumerai SB. 1998 Shifting physician prescribing to a preferred histamine-2-receptor antagonist. Effects of a multifactorial intervention in a mixed-model health maintenance organization. Med. Care 36: 321- 332.
Campbell DT, Stanley JC. 1966 Experimental and quasi-experimental designs for research. Chicago: Rand McNally
Clarke JA, Adams JE. 1990 The application of clinical guidelines for skull radiography in the accident and emergency department: Theory and practice. Clin Radiol. 41: 152-5.
Cochrane AL. 1989 Effectiveness and efficiency: Random reflections on health services. London: British Medical Journal for Nuffield Provincial Health Trust
Cook TD, Campbell DT. 1979 Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton Mifflin Company
Crosbie 1993 J. Interrupted time-series analysis with brief single-subject data. J Consult Clin Psychol. 61: 966- 974.
Crosbie J. 1995 Interrupted time-series analysis with short series; why it is problematic; how it can be improved. In: Gottman JM, ed. The analysis of change, New Jersey: Lawrence Erlbaum Associates
Del Mar CB, Green AC, Battistutta D. 1997 Do public media campaigns designed to increase skin cancer awareness result in increased skin excision rates? Aust N Z J Public Health. 21: 751- 754.
Dempsey CL. 1995 Nursing home-acquired pneumonia: Outcomes from a clinical process improvement program. Pharmacotherapy. 15: 33S- 38S.
1989 Department of Health and the Welsh Office. General practice in the National Health Service: A new contract. London: HMSO
Domenighetti G, Luraschi P, Casabianca A et al. 1988 Effect of information campaign by the mass media on hysterectomy rates. Lancet. 2: 1470- 1473.
Draper N, Smith H. 1981 Applied regression analysis. New York: Wiley
Elam K, Taylor V, Ciol MA, Franklin GM, Deyo RA. 1997 Impact of a worker's compensation practice guideline on lumbar spine fusion in Washington State. Med Care. 35: 417- 424.
Eppler E, Eisenberg MS, Schaeffer S, Meischke H, Larson MP. 1994 911 and emergency department use for chest pain: Results of a media campaign. Ann Emerg Med. 24: 202- 208.
Everitt DE, Soumerai SB, Avorn J, Klapholz H, Wessels M. 1990 Changing surgical antimicrobial prophylaxis practices through education targeted at senior department leaders. Infect Control Hosp Epidemiol. 11: 578- 583.
Fowkes FG, Evans RC, Williams LA et al. 1984 Implementation of guidelines for the use of skull radiographs in patients with head injuries. Lancet. 2: 795- 796.
Fowkes FG, Hall R, Jones JH et al. 1986 Trial of strategy for reducing the use of laboratory tests. BMJ Clin Res Ed. 292: 883- 885.
Fraser GL, Wennberg DE, Dickens JD Jr, Lambrew CT. 1996 Changing physician behavior in ordering digoxin assays. Ann Pharmacother 30: 449- 454.
Gama R, Nightingale PG, Ratcliffe JG. 1992 Effect of educational feedback on clinicians' requesting of cardiac enzymes. Ann Clin Biochem. 29: 224- 225.
Gortmaker SL, Bickford AF, Mathewson HO, Dumbaugh K Tirrell PC. 1988 A successful experiment to reduce unnecessary laboratory use in a community hospital. Med Care. 26: 631- 642.
Gottman JM. 1981 Time series analysis: A comprehensive introduction for social scientists. Cambridge, England: Cambridge University Press
Grilli R, Ramsay CR, Minozzi S. 2002 Mass media interventions: Effects on health services utilisation. In: The Cochrane Library. Oxford: Update Software Issue 1.
Grimshaw JM, Thomas RE, MacLennan G et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess (in press).
Gurwitz JH, Noonan JP, Soumerai SB. 1992 Reducing the use of H2-receptor antagonists in the long-term-care setting. J Am Geriatr Soc. 40: 359- 364.
Hammond KW, Snowden M, Risse SC, Adkins TG, O'Brien JJ. 1995 An effective computer-based tardive dyskinesia monitoring system. Am J Med Qual. 10: 133- 137.
Harris DW, Karandikar DS, Spencer MG et al. 1979 Returned-medicines campaign in Birmingham, 1977. Lancet. 1: 599- 601.
Healsmith MF, Graham-Brown RA, Osborne JE, London SP, Fletcher A. 1993 Further experience of public education for the early diagnosis of malignant melanoma in Leicestershire. Clin Exp Dermatol. 18: 396- 400.
Herd RM, Cooper EJ, Hunter JA et al. 1995 Cutaneous malignant melanoma. Publicity screening clinics and survival – the Edinburgh experience 1982-90. Br J Dermatol. 132: 563- 570.
Joshi UY, Cameron SO, Sommerville JM, Sommerville RG. 1988 HIV testing in Glasgow Genito-Urinary Medicine Clinics 1985-1987. Scott Med J. 33: 294- 295.
Kong KM, Johnson WF, Anderson PO, Sato LA, Simonian RI. 1987 Effect of a pharmacy-based cost-containment program on cimetidine costs and use. Am J Hosp Pharm. 44: 2068- 2073.
Legorreta AP, Hasan MM, Peters AL, Pelletier KR, Leung KM. 1997 An intervention for enhancing compliance with screening recommendations for diabetic retinopathy. A bicoastal experience. Diabetes Care 20: 520- 523.
Lomas J, Anderson GM, Domnick-Pierre K et al. 1989 Do practice guidelines guide practice? The effect of a consensus statement on the practice of physicians. N Engl J Med. 321: 1306- 1311.
Lowe JB, Balanda KP, Del Mar CB, Purdie D, Hilsdon AM. 1994 General practitioner and patient response during a public education program to encourage skin examinations. Med J Aust. 161: 195- 198.
Macdonald H, Roder D. 1985 The planning, implementation and evaluation of an immunization promotion campaign in South Australia. Hygie. 4: 13- 17.
Maclure M, Dormuth C, Naumann T et al. 1998 Influences of educational interventions and adverse news about calcium-channel blockers on first-line prescribing of antihypertensive drugs to elderly people in British Columbia. Lancet. 352: 943- 948.
Matowe L, Ramsay CR, Grimshaw JM, Gilbert FI, Macleod MJ, Needham G. 2002 Effects of mailed dissemination of the Royal College of Radiologists' guidelines on general practitioner referrals for radiography: A time series analysis. Clin Radiol. 57: 575- 578.
Morrison JC, Sumrall DD, Chevalier SP et al. 1993 The effect of provider education on blood utilization practices. Am J Obstet Gynecol 169: 1240- 1245.
Nattinger AB, Hoffmann RG, Howell-Pelz A, Goodwin JS. 1998 Effect of Nancy Reagan's mastectomy on choice of surgery for breast cancer by US women. JAMA. 279: 762- 766.
Novich M, Gillis L, Tauber AI. 1985 The laboratory test justified. An effective means to reduce routine laboratory testing. Am J Clin Pathol. 84: 756- 759.
Paunio M, Virtanen M, Peltola H et al. 1991 Increase of vaccination coverage by mass media and individual approach: Intensified measles, mumps, and rubella prevention program in Finland. Am J Epidemiol. 133: 1152- 1160.
Pearce MJ, Begg EJ. 1997 Encouraging consensus cost effective drug therapy: Five years experience with a hospital drug utilisation review programme. N Z Med J. 110: 92- 95.
Pehamberger H, Binder M, Knollmayer S, Wolff K. 1993 Immediate effects of a public education campaign on prognostic features of melanoma. J Am Acad Dermatol. 29: 106- 109.
Poma PA. 1998 Effect of departmental policies on cesarean delivery rates: A community hospital experience. Obstet Gynecol 91: 1013- 1018.
Ratnaike S, Hunt D, Eilermann LJ, Hazen R, Deam D. 1993 The investigation of chest pain: Audit and intervention. Med J Aust. 159: 666- 671.
1993 Royal College of Radiologists Working Party. Influence of Royal College of Radiologists' guidelines on referral from general practice. BMJ. 306: 110- 111.
Santerre RE. 1996 The effect of the ACOG guideline on vaginal births after cesarean. Med Care Res Rev. 53: 315- 329.
1989 Secretaries of State for Social Services WNIaS. Working for patients. London: HMSO
Shelley JM, Irwig LM, Simpson JM, Makaskill P. 1991 Evaluation of a mass-media-led campaign to increase Pap smear screening. Health Educ Res. 6: 267- 277.
Sherman CR, Potosky AL, Weis KA, Ferguson JH. 1992 The Consensus Development Program. Detecting changes in medical practice following a consensus conference on the treatment of prostate cancer. Int J Technol Assess Health Care. 8: 683- 693.
Shorr RI, Fought RL, Ray WA. 1994 Changes in antipsychotic drug use in nursing homes during implementation of the OBRA-87 regulations. JAMA. 271: 358- 362.
Soumerai SB, Ross-Degnan D, Kahn JS. 1992 Effects of professional and media warnings about the association between aspirin use in children and Reye's syndrome. Milbank Q. 70: 155- 182.
Soumerai SB, Avorn J, Gortmaker S, Hawley S. 1987 Effect of government and commercial warnings on reducing prescription misuse: The case of propoxyphene. Am J Public Health. 77: 1518- 1523.
Stuart ME, Macuiba J, Heidrich F et al. 1997 Successful implementation of an evidence-based clinical practice guideline: Acute dysuria/urgency in adult women. HMO Pract. 11: 150- 157.
Studnicki J, Remmel R, Campbell R, Werner DC. 1997 The impact of legislatively imposed practice guidelines on cesarean section rates: The Florida experience. Am J Med Qual. 12: 62- 68.
Suwangool P, Moola-Or P, Waiwatana A et al. 1991 Effect of a selective restriction policy on antibiotic expenditure and use: An institutional model. J Med Assoc Thai. 74: 272- 275.
Tesoriero JM, Sorin MD. 1992 The effect of ‘Magic’ Johnson's HIV disclosure on anonymous HIV counselling and testing services in New York State. AIDS Public Policy J. 7: 216- 224.
Thamer M, Ray NF, Henderson SC et al. 1998 Influence of the NIH Consensus Conference on Helicobacter pylori on physician prescribing among a Medicaid population. Med Care. 36: 646- 660.
Turner GC, Mutton KJ. 1987 HIV testing: Changing trends. BMJ. 295: 502.
van Walraven C, Goel V, Chan B. 1998 Effect of population-based interventions on laboratory utilization: A time-series analysis. JAMA. 280: 2028- 2033.
Vincent EC, Hardin PA, Norman LA, Lester EA, Stinton SH. 1995 The effects of a computer-assisted reminder system on patient compliance with recommended health maintenance procedures. Proc Ann Symp Comput Appl Med Care. 656- 660.
Wong ET, McCarron MM. Shaw ST Jr. 1983 Ordering of laboratory tests in a teaching hospital. Can it be improved? JAMA. 249: 3076- 3080.
Zehr KJ, Dawson PB, Yang SC, Heitmiller RF. 1998 Standardized clinical care pathways for major thoracic cases reduce hospital costs. Ann Thorac Surg. 66: 914- 919.


Related content

Powered by UNSILO


  • Craig R. Ramsay (a1), Lloyd Matowe (a2), Roberto Grilli (a3), Jeremy M. Grimshaw (a4) and Ruth E. Thomas (a1)...


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed.