Skip to main content Accessibility help

Interrater Reliability of Surveillance for Ventilator-Associated Events and Pneumonia

  • Meeta Prasad Kerlin (a1), William E. Trick (a2), Deverick J. Anderson (a3), Hilary M. Babcock (a4), Ebbing Lautenbach (a1), Renaud Gueret (a2) and Michael Klompas (a5) (a6)...



To compare interrater reliabilities for ventilator-associated event (VAE) surveillance, traditional ventilator-associated pneumonia (VAP) surveillance, and clinical diagnosis of VAP by intensivists.


A retrospective study nested within a prospective multicenter quality improvement study.


Intensive care units (ICUs) within 5 hospitals of the Centers for Disease Control and Prevention Epicenters.


Patients who underwent mechanical ventilation.


We selected 150 charts for review, including all VAEs and traditionally defined VAPs identified during the primary study and randomly selected charts of patients without VAEs or VAPs. Each chart was independently reviewed by 2 research assistants (RAs) for VAEs, 2 hospital infection preventionists (IPs) for traditionally defined VAP, and 2 intensivists for any episodes of pulmonary deterioration. We calculated interrater agreement using κ estimates.


The 150 selected episodes spanned 2,500 ventilator days. In total, 93–96 VAEs were identified by RAs; 31–49 VAPs were identified by IPs, and 29–35 VAPs were diagnosed by intensivists. Interrater reliability between RAs for VAEs was high (κ, 0.71; 95% CI, 0.59–0.81). Agreement between IPs using traditional VAP criteria was slight (κ, 0.12; 95% CI, −0.05–0.29). Agreement between intensivists was slight regarding episodes of pulmonary deterioration (κ 0.22; 95% CI, 0.05–0.39) and was fair regarding whether episodes of deterioration were attributable to clinically defined VAP (κ, 0.34; 95% CI, 0.17–0.51). The clinical correlation between VAE surveillance and intensivists’ clinical assessments was poor.


Prospective surveillance using VAE criteria is more reliable than traditional VAP surveillance and clinical VAP diagnosis; the correlation between VAEs and clinically recognized pulmonary deterioration is poor.

Infect Control Hosp Epidemiol 2017;38:172–178


Corresponding author

Address correspondence to Meeta Prasad Kerlin, Perelman School of Medicine at the University of Pennsylvania, 3600 Spruce St, Gibson 5011, Philadelphia, PA 19104 (


Hide All
1. Safdar, N, Dezfulian, C, Collard, HR, Saint, S. Clinical and economic consequences of ventilator-associated pneumonia: a systematic review. Crit Care Med 2005;33:21842193.
2. Klompas, M. Interobserver variability in ventilator-associated pneumonia surveillance. Am J Infect Control 2010;38:237239.
3. Stevens, JP, Kachniarz, B, Wright, SB, et al. When policy gets it right: variability in US hospitals’ diagnosis of ventilator-associated pneumonia. Crit Care Med 2014;42:497503.
4. Surveillance for Ventilator-Associated Events. Centers for Disease Control and Prevention website. Published 2015. Accessed March 31, 2016.
5. Klein Klouwenberg, PM, van Mourik, MS, Ong, DS, et al. Electronic implementation of a novel surveillance paradigm for ventilator-associated events: feasibility and validation. Am J Respir Cri. Care Med 2014;189:947955.
6. Mann, T, Ellsworth, J, Huda, N, et al. Building and validating a computerized algorithm for surveillance of ventilator-associated events. Infect Control Hosp Epidemiol 2015:15.
7. Klompas, M, Anderson, D, Trick, W, et al. The preventability of ventilator-associated events. The CDC Prevention Epicenters Wake Up and Breathe Collaborative. Am J Respir Crit Care Med 2015;191:292301.
8. Horan, TC, Andrus, M, Dudeck, MA. CDC/NHSN surveillance definition of health care–associated infection and criteria for specific types of infections in the acute care setting. Am J Infect Control 2008;36:309332.
9. Magill, SS, Klompas, M, Balk, R, et al. Developing a new, national approach to surveillance for ventilator-associated events. Crit Care Med 2013;41:24672475.
10. Cohen, J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:3746.
11. Landis, JR, Koch, GG. The measurement of observer agreement for categorical data. Biometrics 1977:159174.
12. Emori, TG, Edwards, JR, Culver, DH, et al. Accuracy of reporting nosocomial infections in intensive-care–unit patients to the National Nosocomial Infections Surveillance System: a pilot study. Infect Cont Hosp Ep 1998;19:308316.
13. Schurink, CA, Van Nieuwenhoven, CA, Jacobs, JA, et al. Clinical pulmonary infection score for ventilator-associated pneumonia: accuracy and inter-observer variability. Intens Care Med 2004;30:217224.
14. Tejerina, E, Esteban, A, Fernández-Segoviano, P, et al. Accuracy of clinical definitions of ventilator-associated pneumonia: comparison with autopsy findings. J Crit Care 2010;25:6268.


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed