Skip to main content Accessibility help
×
Home

Information:

  • Access
  • Cited by 85

Actions:

      • Send article to Kindle

        To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Quantifying the Hawthorne Effect in Hand Hygiene Compliance Through Comparing Direct Observation With Automated Hand Hygiene Monitoring
        Available formats
        ×

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Quantifying the Hawthorne Effect in Hand Hygiene Compliance Through Comparing Direct Observation With Automated Hand Hygiene Monitoring
        Available formats
        ×

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Quantifying the Hawthorne Effect in Hand Hygiene Compliance Through Comparing Direct Observation With Automated Hand Hygiene Monitoring
        Available formats
        ×
Export citation

Abstract

OBJECTIVE

To quantify the Hawthorne effect of hand hygiene performance among healthcare workers using direct observation.

DESIGN

Prospective observational study.

SETTING

Intensive care unit, university hospital.

METHODS

Direct observation of hand hygiene compliance over 48 audits of 2 hours each. Simultaneously, hand hygiene events (HHEs) were recorded using electronic alcohol-based handrub dispensers. Directly observed and electronically recorded HHEs during the 2 hours of direct observation were compared using Spearman correlations and Bland-Altman plots. To quantify the Hawthorne effect, we compared the number of electronically recorded HHEs during the direct observation periods with the re-scaled electronically recorded HHEs in the 6 remaining hours of the 8-hour working shift.

RESULTS

A total of 3,978 opportunities for hand hygiene were observed during the 96 hours of direct observation. Hand hygiene compliance was 51% (95% CI, 49%–53%). There was a strong positive correlation between directly observed compliance and electronically recorded HHEs (ρ=0.68 [95% CI, 0.49–0.81], P<.0001). In the 384 hours under surveillance, 4,180 HHEs were recorded by the electronic dispensers. Of those, 2,029 HHEs were recorded during the 96 hours in which direct observation was also performed, and 2,151 HHEs were performed in the remaining 288 hours of the same working shift that were not under direct observation. Healthcare workers performed 8 HHEs per hour when not under observation compared with 21 HHEs per hour during observation.

CONCLUSIONS

Directly and electronically observed HHEs were in agreement. We observed a marked influence of the Hawthorne effect on hand hygiene performance.

Infect Control Hosp Epidemiol 2015;36(8):957–962

Hand hygiene is universally acknowledged to be one of the most important strategies to prevent healthcare-associated infections and the transmission of multidrug-resistant organisms, which both cause significant morbidity and mortality among hospitalized patients. 1 3 Nevertheless, adherence to recommended hand hygiene practices remains low. 4 The main methods for hand hygiene adherence assessments are direct observations, indirect assessments by measuring the volume of consumed disinfectants or counting dispenser utilizations, and advanced automated adherence monitoring technology. Direct observation, in which the observers monitor the hand hygiene compliance of healthcare workers (HCWs) during their clinical tasks, is the most commonly used method and considered to be the gold standard. 2 However, direct observation of HCWs is prone to the Hawthorne effect, which refers to the tendency of people to behave differently when they know that they are being observed. 5 , 6 Proposed explanations for the Hawthorne effect include the psychological effect of being singled out, noticed, or made to feel important. 7 The Hawthorne effect is widely assumed to increase the hand hygiene compliance rates of HCWs when observers are present. However, only a few studies have examined the actual magnitude of the effect and only a few lacked methodologic weaknesses, including selection and observer biases. 8 15 The recent development of electronic systems for hand hygiene monitoring offers an alternative approach to direct observation and might overcome the aforementioned obstacles. 5 These electronic systems monitor hand hygiene on a constant, real-time basis without requiring direct observation, hence making them a promising tool for reevaluation of the Hawthorne effect. Therefore, the objective of the study was to quantify the Hawthorne effect on direct hand hygiene compliance observations by means of electronic hand hygiene monitoring.

Methods

The study was performed in a 24-bed anesthesiological-surgical intensive care unit (ICU) at the University Hospital of Jena, a 1,500-bed tertiary care hospital, from October 1, 2013, through February 28, 2014. The ICU consisted of five 4-bed rooms, one 2-bed room, and two single-bed rooms. The study was approved by the facility’s institutional review board.

Electronic Hand Hygiene Dispensers

Overall, 70 electronic hand hygiene dispensers (ingo-man Weco, Ophardt) of alcohol-based handrub (AHR) were installed in the ICU in January 2013. Each of the 4-bed rooms was equipped with 7 to 11 dispensers located at each bedside, at the sink, and at the bench. Each time the lever of the dispenser is pushed, a time-stamped hand hygiene event (HHE) is recorded and subsequently transmitted via WiFi to a centralized database. The dispenser records only 1 HHE in any 2-second period even if the lever is pushed more than once. The energy for the WiFi data transfer is obtained by pushing the lever of the dispenser and it does not require any additional power supply. Other sources of AHR (eg, pocket bottles) were not commonly used. HCWs were aware that the electronic dispensers were used to monitor hand hygiene.

Direct Observation

Since 2012, direct observations of hand hygiene adherence measurements have been regularly performed by infection control practitioners. The last observation in summer 2013 in the ICU revealed an overall hand hygiene compliance of 41% according to the World Health Organization’s (WHO) “Five Moments for Hand Hygiene.” 16 For this study, direct hand hygiene observations were performed by a medical student intern observer (J.R.) who was trained and validated in hand hygiene observation techniques by infection control practitioners. Fifty total episodes of direct observations were performed within the study, each lasting 2 hours. During the entire observation period, the observer stayed in the ward room and counted all HHEs and opportunities in accordance with the WHO’s “Five Moments for Hand Hygiene.” When 2 hand hygiene opportunities occurred at a single time point, the procedure that theoretically had a greater impact was recorded (eg, category 2 [before clean/aseptic procedure] rather than category 1 [before touching a patient]). When 2 consecutive hand hygiene opportunities occurred (eg, HCW leaves a patient and goes to next patient), only the first event was documented to avoid ‘‘double-counting’’ discrepancies. Applying and changing gloves without the use of AHR or glove disinfection was not counted as a proper HHE. Hand hygiene technique, such as the amount of time spent using an AHR and the extent to which all surfaces of the hands are covered, was not evaluated. To achieve an overall high number of HHEs, we decided to limit the observations to the 4-bed rooms only. Every HCW (eg, nurses, physicians, x-ray technicians, physiotherapists) within the patient room was observed. No feedback was given to the HCWs. When starting the observation, the observer introduced herself and the purpose of the observation (“hand hygiene compliance observation”) to the attending nurses (n=2) who were responsible for patient care in the observed 4-bed room. However, the primary objective of the study, quantifying the Hawthorne effect, was not communicated. The observer did not personally inform each additional HCW who entered the patient room about the purpose of the observation but did so when actively asked. All observations took place between Mondays and Fridays between 6:00 AM and 10:00 PM—that is, during 2 working shifts (morning shift 6:00 AM–2:00 PM and afternoon shift 2:00 PM–10:00 PM). The time of observation and the patient room were randomly selected by the observer at the day of the observation.

Statistical Analyses

Hand hygiene compliance was calculated as the number of observed HHEs according to the WHO’s “Five Moments for Hand Hygiene” divided by the total number of opportunities observed, expressed as a percentage. The agreement between the number of HHEs that were directly observed and the number that were electronically recorded was analyzed using a Bland-Altman plot that compared the mean of the values provided by the 2 methods and their differences. The Hawthorne effect was investigated by comparing the number of the electronically recorded HHEs in the observation period with those in the remaining 6 hours of the working shift. This value was standardized to a 2-hour interval to allow for comparability. We quantified the Hawthorne effect as the explained variability (R 2) of HHEs/2 hours (quantitative outcome) given the presence or absence of an observer using a general linear model. To describe the monotonic relationship between the 2 quantitative variables, we used the Spearman rank correlation coefficient ρ with 95% CIs. We applied a 2-sided significance level of 5% and reported 2-sided P values that are not corrected for multiple testing. The data were analyzed using the statistical language R, version 3.0.3 (R Foundation for Statistical Computing), and SAS, version 9.3 (SAS Institute).

Results

In total, 50 episodes of direct hand hygiene observation were performed, each lasting 2 hours. During 2 episodes, 1 electronic dispenser was broken, and the data were not transmitted to the centralized database. As a result, these 2 episodes were excluded from further analyses. Overall, 96 hours of direct observation and 288 hours without direct observation were included in the analyses. Apart from 8 observations with 1 vacant bed, all beds in the observed 4-bed rooms were occupied. Thus, there were 184 total patients included throughout the whole study period. Twenty-seven of the observations (56%) were performed during the morning shift, and 21 (44%) were performed during the afternoon shift.

Hand Hygiene Compliance

In total, 4,077 HHEs were observed during the 96 hours of direct observation. Of those, 3,978 events were considered to be hand hygiene opportunities according to the recommendations of the WHO’s “Five Moments for Hand Hygiene” (ie, 3,978 hand hygiene opportunities in 184 patients each 2-hour observation, or 11 opportunities for hand hygiene per patient per hour). Ninety-nine HHEs that were performed were not events that met the WHO recommendations. The overall rate of observed hand hygiene compliance was 51% (95% CI, 49%–53%); in 2,034 of the 3,978 opportunities, a hand hygiene performance was observed. Detailed compliance rates by specific opportunities and the profession of the HCWs are displayed in Tables 1 and 2. Next, we explored the Spearman correlations between hand hygiene compliance according to the WHO standards and directly observed HHEs (ρ=0.69 [95% CI 0.50–0.81], P<.0001) as well as electronically recorded HHEs during direct observation (ρ=0.68 [95% CI, 0.49–0.81], P<.0001). In both cases, we observed a strong monotonic relationship (Figure 1).

FIGURE 1 A, Hand hygiene compliance versus directly recorded hand hygiene events (HHEs) during the 2-hour direct observation period. B, Hand hygiene compliance versus electronically recorded HHEs during the 2-hour direct observation period.

TABLE 1 Compliance With Hand Hygiene According to Indication

TABLE 2 Compliance With Hand Hygiene According to Profession

Agreement Between Direct Observation and the Electronic Count System

The Bland-Altman plot indicates a good agreement between the average number of HHEs captured with direct observation and the average number of HHEs registered by the electronic dispensers during the same 2-hour period. The mean difference between both methods was 2 HHEs per 2 hours of observation (95% CI, -7–11). On average, 2 additional HHEs were recorded in each 2-hour period with direct observation compared with electronic recording (Figure 2).

FIGURE 2 Bland-Altman plot of the agreement between the average number of hand hygiene events in 2 hours assessed by the following 2 methods: direct observations or the electronic count system. In total, 48 episodes (each dot is 1 episode) are displayed. The horizontal lines are the average difference of the methods and corresponding 95% CI. obs., observation.

Hawthorne Effect

In total, 4,180 HHEs were recorded by the electronic dispensers over 384 hours of the 48 analyzed working shifts. Of those, 2,029 HHEs were simultaneously recorded with the 96 hours of direct hand hygiene observation, and 2,151 HHEs were recorded in the remaining 288 hours of the same working shift that did not have direct hand hygiene observation. The mean difference in the number of HHEs per 2-hour period between episodes of direct hand hygiene observation and episodes without observation was 27 HHEs (95% CI, 23–31 HHEs) (Figure 3). Healthcare workers performed 8 HHEs per hour when not under observation compared with 21 HHEs per hour during observation. This figure translates into 5 HHEs per patient per hour under observation and 2 HHEs per patient per hour in the periods without observation. In a general linear model with HHEs as the quantitative outcome, 61% of the observed total HHE variability was explained by the presence or absence of a direct observer (ie, the Hawthorne effect).

FIGURE 3 Box plot of the differences in hand hygiene events (HHEs) between the episodes under direct observation and the episodes not under observation in a 2-hour period of the electronic count system. obs., observation.

Discussion

Our study demonstrates the marked influence of the Hawthorne effect on hand hygiene performance. We observed an average increase from 8 HHEs per hour without direct observation to 21 HHEs per hour under direct observation in a 4-bed room, which corresponds to 2 HHEs and 5 HHEs per patient per hour, respectively. This result confirms the findings of the studies addressing the Hawthorne effect on hand hygiene performance that have been published so far. Srigley et al 14 likewise observed an approximately 3-fold increase of HHEs in the hallways when the auditors were visible compared with when they were not. Cheng et al 17 used an electronic monitoring system to quantify the Hawthorne effect and observed an approximately 180% increase in hand hygiene compliance during audits (32% vs 89%). Eckmanns et al, 9 however, reported a less pronounced increase of 55% in compliance between covert and overt observations in 5 German ICUs (29% vs 45%). Whether differing baseline levels of hand hygiene compliance determines the impact of known observers on hand hygiene performance was assessed by Kohli et al. 11 They demonstrated that the Hawthorne effect was more pronounced in units with a high baseline level (79% without observation vs 98% with observation) than in units with a low baseline level (40% without observation vs 47% with observation). Because the Hawthorne effect varies with different baseline levels, it is therefore unfeasible to generally estimate the true compliance of hand hygiene by merely dividing the observed rate by 3, assuming a 3-fold increase under observation as recorded in our study. To minimize the Hawthorne effect, some facilities have used covert observers or “secret shoppers,” which may improve the validity of the measurements. 2 However, some experts have raised ethical concerns about not receiving informed consent of those being observed, and it is unlikely that the covert nature of the observation can be sustained over a long period. 15 Thus, there are a number of studies that have undeniably shown the marked influence of the Hawthorne effect during direct hand hygiene observation. In contrast to the aforementioned studies, however, our results are not limited to a specific indication or location (eg, hallways). Furthermore, our study estimated the Hawthorne effect by using electronic dispensers in contrast to studies that relied on observers unknown to HCWs, who also might themselves provoke the Hawthorne effect.

In addition to the biased estimates due to the Hawthorne effect, direct human observation is labor intensive and costly. Furthermore, direct human observations are subject to observer and selection biases and often provide information for only a small sample (1%–3%) of all hand hygiene opportunities occurring in healthcare settings. 5 Conclusions about an entire ward, department, or profession may thus be subject to both systematic and random errors. Automated hand hygiene monitoring technologies could improve methods for hand hygiene adherence measurement. They can record much greater numbers of HHEs than can be achieved by direct observation while also avoiding a Hawthorne effect or observer bias, and they can provide specific data on the number of HHEs occurring per patient-day. 2 The results of our study demonstrate that hand hygiene compliance may be measurable without requiring direct human observation. Prerequisite for using automated hand hygiene monitoring technology in hand hygiene adherence measurements is the concordance between the number of electronically recorded and the directly observed HHEs. We observed no evidence to assume that the number of directly observed and electronically recorded HHEs substantially differed. Additionally, our study revealed a strong correlation between the electronically recorded HHEs and the directly observed compliance rate. During direct observation, we recorded approximately 11 average necessary HHEs per patient per hour according to the recommendations of the WHO. This number is in line with the results of a study by Scheithauer et al 18 who observed 10 necessary HHEs per patient per hour in the morning shift and 8 HHEs per patient per hour in the afternoon shift. Based on these 11 necessary HHEs per patient per hour, 5 HHEs per patient per hour translates to a compliance of 45%, which is close to the directly observed compliance of 51%. In contrast, only 2 HHEs per patient per hour were performed without observation, which translates to a compliance of less than 20%. One point often highlighted is the fact that electronic counting devices cannot determine whether HHEs are performed at appropriate moments (ie, the WHO’s “Five Moments for Hand Hygiene”), and an estimation of hand hygiene compliance rates would therefore not be correct. However, we could show for the first time that during the whole period of direct observation of 2,133 total HHEs, less than 5% (n=99) were not indicated according to the recommendations of the WHO’s “Five Moments for Hand Hygiene.” The bias of unindicated HHEs seems to be negligible, at least for the ICU setting with a high density of opportunities for hand hygiene.

However, several limitations of electronic counting devices must be considered. These limitations include the inability to distinguish between product use by HCWs versus patients and visitors, the inability to identify specific HCWs or type of HCW using the AHR, and the inability to evaluate hand hygiene techniques or the identification of specific issues that warrant further education of HCWs. 5 Furthermore, they have upfront costs related to the installation of new hardware and maintenance costs. In contrast, direct observation can discern all opportunities for hand hygiene within patient care encounters, can provide detailed information about hand hygiene technique and compliance rates for HCWs of different types and levels of seniority, and can identify specific situations that require further education of HCWs. 2 , 5 Above all, direct observation allows for direct individual performance feedback (ie, “just in time training”), making the observation part of an intervention.

Limitations of the Study

The generalizability of our findings is unknown because our study was limited to 1 ICU in a single healthcare facility. However, we have no reason to believe that our general conclusions will be different in other settings. Furthermore, we could not match with individual HHEs and had to work with summary measures during a certain period. The same is true when trying to differentiate the HHEs between individual HCWs given that no individual HCW identification was recorded. Additionally, we did not evaluate the hand hygiene technique and did not include HCWs within the night shift, who might have responded differently to the observation. In addition, we cannot rule out that visitors may also have used the electronic dispensers, which may have led to an overestimation of the number of HHEs performed by the HCWs. Finally, we could not control for the possibility of a variable workload of the HCWs during the different observed episodes, including with and without direct observation.

In summary, our study adds to other studies demonstrating the impact of the Hawthorne effect on hand hygiene adherence in HCWs. Because many healthcare facilities use infection control practitioners as observers of hand hygiene adherence, it is important to recognize that this practice most likely leads to overestimation of the true rate of hand hygiene compliance. Furthermore, our study demonstrates a good agreement between directly and electronically observed HHEs and reveals a strong correlation between the electronically recorded HHEs and the directly observed compliance rate, hence making electronic counting devices a promising tool for assessing hand hygiene compliance. For the future, we envision a combined approach of direct observation without measuring compliance, giving only individual performance feedback, and using automated adherence monitoring to realistically estimate hand hygiene adherence while minimizing observer bias and the Hawthorne effect.

Acknowledgments

Financial support. German Ministry of Education and Research (grants 01KI1204 and 01EO1002). The electronic hand hygiene dispensers were provided for free by Ophardt.

Potential conflicts of interest. S.H. reports that he received travel grants from Ophardt. All other authors report no conflicts of interest relevant to this article.

Authorship and manuscript preparation. Ophardt had no role in the gathering or preparation of data or in the writing of the manuscript.

References

1. Barnes, SL, Morgan, DJ, Harris, AD, Carling, PC, Thom, KA. Preventing the transmission of multidrug-resistant organisms: modeling the relative importance of hand hygiene and environmental cleaning interventions. Infect Control Hosp Epidemiol 2014;35:11561162.
2. Ellingson, K, Haas, JP, Aiello, AE, et al. Strategies to prevent healthcare-associated infections through hand hygiene. Infect Control Hosp Epidemiol 2014;35:937960.
3. Hagel, S, Ludewig, K, Frosinski, J, et al. Effectiveness of a hospital-wide educational programme for infection control to reduce the rate of health-care associated infections and related sepsis (ALERTS)—methods and interim results. Dtsch Med Wochenschr 2013;138:17171722.
4. Erasmus, V, Daha, TJ, Brug, H, et al. Systematic review of studies on compliance with hand hygiene guidelines in hospital care. Infect Control Hosp Epidemiol 2010;31:283294.
5. Boyce, JM. Measuring healthcare worker hand hygiene activity: current practices and emerging technologies. Infect Control Hosp Epidemiol 2011;32:10161028.
6. Haas, JP, Larson, EL. Measurement of compliance with hand hygiene. J Hosp Infect 2007;66:614.
7. Holden, JD. Hawthorne effects and research into professional practice. J Eval Clin Pract 2001;7:6570.
8. Dhar, S, Tansek, R, Toftey, EA, et al. Observer bias in hand hygiene compliance reporting. Infect Control Hosp Epidemiol 2010;31:869870.
9. Eckmanns, T, Bessert, J, Behnke, M, Gastmeier, P, Ruden, H. Compliance with antiseptic hand rub use in intensive care units: the Hawthorne effect. Infect Control Hosp Epidemiol 2006;27:931934.
10. Harbarth, S, Pittet, D, Grady, L, et al. Interventional study to evaluate the impact of an alcohol-based hand gel in improving hand hygiene compliance. Pediatr Infect Dis J 2002;21:489495.
11. Kohli, E, Ptak, J, Smith, R, Taylor, E, Talbot, EA, Kirkland, KB. Variability in the Hawthorne effect with regard to hand hygiene performance in high- and low-performing inpatient care units. Infect Control Hosp Epidemiol 2009;30:222225.
12. Maury, E, Moussa, N, Lakermi, C, Barbut, F, Offenstadt, G. Compliance of health care workers to hand hygiene: awareness of being observed is important. Intensive Care Med 2006;32:20882089.
13. Pan, S-C, Tien, K-L, Hung, I-C, et al. Compliance of health care workers with hand hygiene practices: independent advantages of overt and covert observers. PLOS ONE 2013;8:e53746.
14. Srigley, JA, Furness, CD, Baker, GR, Gardam, M. Quantification of the Hawthorne effect in hand hygiene compliance monitoring using an electronic monitoring system: a retrospective cohort study. BMJ Qual Saf 2014;23:974980.
15. Gould, DJ, Chudleigh, J, Drey, NS, Moralejo, D. Measuring handwashing performance in health service audits and research studies. J Hosp Infect 2007;66:109115.
16. Sax, H, Allegranzi, B, Uçkay, I, Larson, E, Boyce, J, Pittet, D. “My Five Moments for Hand Hygiene”: a user-centred design approach to understand, train, monitor and report hand hygiene. J Hosp Infect 2007;67:921.
17. Cheng, VCC, Tai, JWM, Ho, SKY, et al. Introduction of an electronic monitoring system for monitoring compliance with moments 1 and 4 of the WHO “My 5 Moments for Hand Hygiene” methodology. BMC Infect Dis 2011;11:151.
18. Scheithauer, S, Haefner, H, Schwanz, T, et al. Compliance with hand hygiene on surgical, medical, and neurologic intensive care units: direct observation versus calculated disinfectant usage. Am J Infect Control 2009;37:835841.