Hostname: page-component-7c8c6479df-fqc5m Total loading time: 0 Render date: 2024-03-17T14:27:41.170Z Has data issue: false hasContentIssue false

Exploratory factor analysis and reliability of the Primary Health Care Engagement (PHCE) Scale in rural and remote nurses: findings from a national survey

Published online by Cambridge University Press:  27 July 2017

Julie G. Kosteniuk*
Affiliation:
Professional Research Associate, Canadian Centre for Health and Safety in Agriculture, College of Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada
Norma J. Stewart
Affiliation:
Professor of Nursing, College of Nursing, University of Saskatchewan, Saskatoon, Saskatchewan, Canada
Chandima P. Karunanayake
Affiliation:
Professional Research Associate, Canadian Centre for Health and Safety in Agriculture, University of Saskatchewan, Saskatoon, Saskatchewan, Canada
Erin C. Wilson
Affiliation:
Assistant Professor, School of Nursing, University of Northern British Columbia, Prince George, British Columbia, Canada
Kelly L. Penz
Affiliation:
Assistant Professor, College of Nursing, University of Saskatchewan, Regina Campus, Regina, Saskatchewan, Canada
Judith C. Kulig
Affiliation:
Professor Emerita, Faculty of Health Sciences, University of Lethbridge, Lethbridge, Alberta, Canada
Kelley Kilpatrick
Affiliation:
Associate Professor, Faculty of Nursing, Université de Montréal and CIUSSS EIM-Maisonneuve-Rosemont Hospital site, Montréal, Québec, Canada
Ruth Martin-Misener
Affiliation:
Professor, School of Nursing, Dalhousie University, Halifax, Nova Scotia, Canada
Debra G. Morgan
Affiliation:
Professor, Canadian Centre for Health and Safety in Agriculture, College of Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada
Martha L.P. MacLeod
Affiliation:
Professor and Knowledge Mobilization Research Chair, School of Nursing, University of Northern British Columbia, Prince George, British Columbia, Canada
*
Correspondence to: Julie G. Kosteniuk, Canadian Centre for Health and Safety in Agriculture, University of Saskatchewan, 104 Clinic Place, PO Box 23, Saskatoon, Saskatchewan, Canada, S7N 2Z4. Email: julie.kosteniuk@usask.ca
Rights & Permissions [Opens in a new window]

Abstract

Aim

The study purpose was to provide evidence of validity for the Primary Health Care Engagement (PHCE) Scale, based on exploratory factor analysis and reliability findings from a large national survey of regulated nurses residing and working in rural and remote Canadian communities.

Background

There are currently no published provider-level instruments to adequately assess delivery of community-based primary health care, relevant to ongoing primary health care (PHC) reform strategies across Canada and elsewhere. The PHCE Scale reflects a contemporary approach that emphasizes community-oriented and community-based elements of PHC delivery.

Methods

Data from the pan-Canadian Nursing Practice in Rural and Remote Canada II (RRNII) survey were used to conduct an exploratory factor analysis and evaluate the internal consistency reliability of the final PHCE Scale.

Findings

The RRNII survey sample included 1587 registered nurses, nurse practitioners, licensed practical nurses, and registered psychiatric nurses residing and working in rural and remote Canada. Exploratory factor analysis identified an eight-factor structure across 28 items overall, and good internal consistency reliability was indicated by an α estimate of 0.89 for the final scale. The final 28-item PHCE Scale includes three of four elements in a contemporary approach to PHC (accessibility/availability, community participation, and intersectoral team) and most community-oriented/based elements of PHC (interdisciplinary collaboration, person-centred, continuity, population orientation, and quality improvement). We recommend additional psychometric testing in a range of health care providers and settings, as the PHCE Scale shows promise as a tool for health care planners and researchers to test interventions and track progress in primary health care reform.

Type
Research
Copyright
© Cambridge University Press 2017 

Introduction

There is no single common definition of primary health care (PHC) in the contemporary literature. Further, the term is often not explicitly defined and is sometimes used synonymously with primary care (Henderson et al., Reference Henderson, Koehne, Verrall, Gebbie and Fuller2014; Canadian Nurses Association, 2015; White, Reference White2015). However, PHC is commonly agreed to broadly address the social determinants of health and offer essential education and health care (eg, health promotion; disease prevention, treatment and management) to communities based on principles of accessibility, equity, community participation, and intersectoral collaboration (Henderson et al., Reference Henderson, Koehne, Verrall, Gebbie and Fuller2014; White, Reference White2015; Baum et al., Reference Baum, Freeman, Sanders, Labonte, Lawless and Javanparast2016).

As a component of PHC, primary care focuses on clinical health care services and medical care provided after first contact between individuals and health care providers. Part of the confusion in terminology stems from advocacy efforts to advance the PHC philosophy, resulting in bifurcation of the term ‘primary health care’. PHC that is ‘professional’ (Levesque et al., Reference Levesque, Haggerty, Burge, Beaulieu, Gass, Pineault and Santor2011), ‘profession-centred’ (White, Reference White2015), or ‘selective’ has come to represent a primary care approach (Baum et al., Reference Baum, Freeman, Sanders, Labonte, Lawless and Javanparast2016), while PHC that is ‘community-based’, ‘societal’, and ‘comprehensive’ signals a contemporary PHC approach founded on the original vision set out in the Alma Ata Declaration on Primary Health Care (World Health Organization, 1978).

In developing the Primary Health Care Engagement (PHCE) Scale for health care providers, we sought to closely align the instrument with a contemporary PHC approach that would be relevant to ongoing PHC reform strategies across Canada (eg, Alberta Health, 2014; Saskatchewan Ministry of Health, 2016) and elsewhere (Liu et al., Reference Liu, Wang, Kong and Cheng2011; Thomas et al., Reference Thomas, Wakerman and Humphreys2014; Galdikiene et al., Reference Galdikiene, Asikainen, Rostila, Geen, Balciunas and Suominen2016). Although other instruments are available to measure primary care delivery performance based on perceptions of health care providers (eg, Schoen et al., Reference Schoen, Osborn and Huynh2006; Dahrouge et al., Reference Dahrouge, Hogg, Russell, Geneau, Kristjansson, Muldoon and Johnston2009), community-based PHC performance has not been well addressed at the provider level. Aside from the PHCE Scale, the Canadian Institute for Health Information (CIHI) PHC Provider Survey (Johnston and Burge, Reference Johnston and Burge2013) is the only other provider-level instrument that specifically targets community-based PHC performance, to our knowledge. Developed to complement the CIHI patient experience and organizational surveys, the CIHI provider survey addresses the PHC principle of intersectoral collaboration but not the principles of accessibility, equity, and community participation. Although these latter attributes may be more appropriately measured at the patient level, it is also worthwhile to gauge the degree of workplace involvement in PHC delivery according to front-line health care providers, given their central role in the actualization of PHC principles.

We were also keen that the new PHCE instrument apply to rural and remote (hereafter rural/remote) communities where effective PHC delivery is highly valued in light of persistent difficulties meeting health and social needs (Thomas et al., Reference Thomas, Wakerman and Humphreys2014; Ward et al., Reference Ward, Buykx, Tham, Kinsman and Humphreys2014). Rural/remote health care providers may work alone or in small multidisciplinary teams with members who are not co-located on a regular basis (MacLeod et al., Reference MacLeod, Kulig, Stewart and Pitblado2004), and thus it is important to make the best use of scarce health human resources in these settings (Ford, Reference Ford2016). Community participation in health service planning is also a policy expectation in rural communities, although meaningful participation can be difficult to operationalize and achieve (Kenny et al., Reference Kenny, Farmer, Dickson-Swift and Hyett2015). For these reasons, we drew on community-oriented (Levesque et al., Reference Levesque, Haggerty, Burge, Beaulieu, Gass, Pineault and Santor2011) and community-based (CIHR, 2015) conceptualizations of PHC in the development of the PHCE Scale. Community-oriented PHC is understood as including ‘… a wide range of professionals who deliver a broad spectrum of health and social services’ (Levesque et al., Reference Levesque, Haggerty, Burge, Beaulieu, Gass, Pineault and Santor2011: 23). Further, community-based PHC is person-centred, population-centred, coordinated, and integrated care delivered by multiple health care providers ‘in a range of community settings’ that covers the continuum from primary prevention to palliative care (CIHR, 2015).

The 40-item PHCE Scale was developed in a three-phase process reported previously (Kosteniuk et al., Reference Kosteniuk, Wilson, Penz, MacLeod, Stewart, Kulig, Karunanayake and Kilpatrick2016); 10 dimensions of community-oriented PHC most relevant to community-based and rural/remote PHC were endorsed by our 16-member research team and 19-member advisory board, collectively representing the 13 Canadian provinces and territories. The 40-item PHCE Scale consisted of 10 subscales: accessibility/availability, equity, community participation, intersectoral team, patient-centred care, continuity, population orientation, interdisciplinary collaboration, comprehensiveness, and quality improvement. Based on initial psychometric testing in a pilot survey of 89 nurses in current practice in Canada with nursing experience in rural/remote communities, the 40-item PHCE Scale (3–5 items per subscale) demonstrated good reliability (α=0.91). Although Cronbach’s α coefficients below 0.70 indicated evidence of low reliability for three subscales (equity, comprehensiveness, and patient-centred care), these were nonetheless retained in the final 40-item version given their theoretical importance to community-oriented PHC delivery in rural/remote communities.

The purpose of the present study was to provide validity evidence for the PHCE Scale, by evaluating the scale’s internal structure based on factor analysis and reliability findings from a large national survey of rural/remote regulated nurses in Canada. Recent years have seen major health care reforms targeting PHC in Canada (Strumpf et al., Reference Strumpf, Levesque, Coyle, Hutchinson, Barnes and Wedel2012) and other countries (Liu et al., Reference Liu, Wang, Kong and Cheng2011; Thomas et al., Reference Thomas, Wakerman and Humphreys2014; Galdikiene et al., Reference Galdikiene, Asikainen, Rostila, Geen, Balciunas and Suominen2016). We anticipate the PHCE Scale will be useful to researchers and health care planners in assessing the level of PHC delivery in health care settings, identifying the degree to which PHC elements are being enacted and the elements that may require intervention, and tracking progress in PHC reform.

Methods

Design

This study uses data from the pan-Canadian study ‘Nursing Practice in Rural and Remote Canada II (RRNII)’, which included a cross-sectional survey with a target sample of 10 072 regulated nurses [registered nurses (RNs), nurse practitioners (NPs), licensed practical nurses, and registered psychiatric nurses] residing in rural/remote Canada. Further details of the RRNII survey are reported elsewhere (MacLeod et al., Reference MacLeod, Stewart, Kulig, Anguish, Andrews, Banner, Garraway, Hanlon, Karunanayake, Kilpatrick, Koren, Kosteniuk, Martin-Misener, Mix, Moffitt, Olynick, Penz, Sluggett, Van Pelt, Wilson and Zimmer2017).

Participants

The RRNII survey population included a stratified systematic sample of regulated nurses in every Canadian province, as well as all regulated nurses working in the Territories and all rural and remote NPs. Eligible nurses practiced in a rural or remote community at the time of the survey, or had been on leave for six months or less. For the purposes of eligibility, communities with a core population of less than 10 000 were considered rural (du Plessis et al., Reference du Plessis, Beshiri, Bollman and Clemenson2001), and northern communities in the Territories (Nunavut, Northwest, and Yukon) were considered remote.

Regardless of their primary place of employment, we chose to include all survey participants in the exploratory factor analysis (EFA) for two reasons. First, community-based PHC is delivered ‘in a range of community settings’ outside of clinics and public health settings (CIHR, 2015). Second, institutional settings such as hospitals have a long history of health and wellness promotion in both urban and rural communities (Olden and Hoffman, Reference Olden and Hoffman2011). Therefore, participants were included whose primary place of employment was a community-based health care site [ie, directly accessible by patients and outside of an inpatient setting (Gibson et al., Reference Gibson, Lisy, Davy, Aromataris, Kite, Lockwood, Ritano, McBride and Brown2015)], hospital/rehabilitation or convalescent centre/integrated facility, nursing home/long-term care facility, or other setting (Table 1).

Table 1 Characteristics of Nursing Practice in Rural and Remote Canada II survey respondents included and excluded from analysis

a Calculated by the χ 2 test unless otherwise noted.

b Mann–Whitney U test.

c Defined as primary health care settings that are directly accessible by patients and outside of an inpatient setting (Gibson et al., Reference Gibson, Lisy, Davy, Aromataris, Kite, Lockwood, Ritano, McBride and Brown2015), includes mental health centre/crisis centre, community health centre, home care agency, public health department/unit, private nursing/self-employed, occupational health, physician’s office/family practice unit or team, nurse practitioner led clinic, and multidisciplinary primary health care clinic.

d Includes hospital, rehabilitation/convalescent centre, and integrated facility (ie, acute and long-term care).

e Includes educational institution, professional association/government, and other.

A total of 3822 participants completed the RRNII survey. Of 10 072 sampled, 450 were ineligible (eg, incorrect addresses, duplicate registrations, retired) and 9622 were eligible, resulting in a response rate of 40% (3822/9622). Of these participants, 1587 completed all 40 items in the PHCE Scale and were included in the factor analysis; 2235 participants completed fewer than 40 items and were excluded to avoid artificially high correlations that may result from imputing missing values. Compared with excluded participants, those included were more highly educated and held higher registration status, and were more likely to work in smaller communities under 10 000 population (Table 1).

Data collection

RRNII survey data were collected from April 2014 to August 2015, with surveys distributed by the RRNII’s research centre at the University of Northern British Columbia as well as by provincial and territorial nursing associations. Participant recruitment relied on the Dillman method (Dillman et al., Reference Dillman, Smyth and Christian2014) to improve the response rate (eg, participants were offered a chance to win an iPad as an incentive). Online and paper survey packages were provided in English and French; non-respondents received a total of four contacts that included a first survey package, first reminder, second reminder, and final survey package.

Measures

Development of the original 40-item PHCE Scale consisted of a literature review and expert consultation, content evaluation and item revision, and testing in a small pilot survey of 89 nurses (Kosteniuk et al., Reference Kosteniuk, Wilson, Penz, MacLeod, Stewart, Kulig, Karunanayake and Kilpatrick2016). Items developed for the original PHCE Scale drew from multiple indicators of quality and performance [World Health Organization, 1986; Flocke, Reference Flocke1997; Shi et al., Reference Shi, Starfield and Xu2001; Canadian Institute for Health Information (CIHI), 2006; Davis et al., Reference Davis, Schoen, Schoenbaum, Doty, Holmgren, Kriss and Shea2007; Bloch et al., Reference Bloch, Rozmovits and Giambrone2011; Levesque et al., Reference Levesque, Haggerty, Burge, Beaulieu, Gass, Pineault and Santor2011; Wong et al., Reference Wong, Browne, Varcoe, Lavoi, Smye, Godwin, Littlejohn and Tu2011; Saskatchewan Ministry of Health, 2012], including nine items that were minimally adapted with permission from a study by Dahrouge et al. (Reference Dahrouge, Hogg, Russell, Geneau, Kristjansson, Muldoon and Johnston2009).

The 27-page RRNII survey employed in the present study consisted of five sections that focused on participants’ individual characteristics, work community, workplace, nursing practice, and aspects of their psychosocial health. The 40-item PHCE Scale included three to five items in each of 10 subscales, with items rated on a five-point scale as 1 (strongly disagree), 2 (disagree), 3 (neutral), 4 (agree), 5 (disagree), or 97 (not applicable). ‘Not applicable’ responses were coded as ‘missing’. Participants were instructed to respond in relation to the primary workplace where they spent most of their work time in the past 12 months, and to the community represented by the catchment area of their primary workplace. After reverse scoring negatively worded items, higher scores for each subscale and the overall scale suggest perceptions of greater workplace engagement in primary health care delivery.

Across the 40 PHCE items, the proportion of ‘not applicable’ responses (1.9–49.2%) was substantially higher than the proportion of true missing responses (3.1–7.7%). Specifically, between 20.1 and 49.2% of participants chose the ‘not applicable’ option in each of eight items across three subscales (accessibility/availability, population orientation, and intersectoral team). Participants whose primary place of employment was a community-based health care setting were more likely to provide a valid response to all of the scale items and also significantly more likely than their counterparts in other settings (ie, hospital, nursing home/long-term care facility, and other) to be included in the present analysis (57.1 versus 35.2%; P<0.001). However, those outside of community-based settings accounted for the majority of included participants (59.9%), demonstrating the wide applicability of the PHCE Scale across health care settings.

Statistical analyses

Initial principal component analysis and descriptive statistics were performed with IBM SPSS Statistics 23 and EFA was conducted with SAS 9.3.

The sample of 1587 participants in the present study exceeded the recommended 5:1 participant-to-item ratio and 200 participant minimum (Howard, Reference Howard2016). The large sample size took into account the small number of items per factor (<5) in the 40-item PHCE scale (Gaskin and Happell, Reference Gaskin and Happell2014) as well as potentially weak item communalities (<0.40) (Costello and Osborne, Reference Costello and Osborne2005), which would indicate that the factor model accounted for a small amount of variance in those items (Tabachnick and Fidell, Reference Tabachnick and Fidell2013). Missing values were not imputed, as imputation with estimated values may result in overfit data and artificially high correlations (Tabachnick and Fidell, Reference Tabachnick and Fidell2013). Instead, listwise deletion was employed to delete every case with one or more missing items.

The Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy and Bartlett’s test of sphericity were first calculated with the 1587 participants to assess whether factor analysis was warranted (Howard, Reference Howard2016). A KMO value of 0.80 or above, on a range of 0–1, indicates that the items share a common factor and factor analysis is supported (Kellar and Kelvin, Reference Kellar and Kelvin2013).

An initial principal component analysis to calculate eigenvalues was conducted, and the number of factors to extract was identified with the assistance of web-based parallel analysis (Patil, 2008). Parallel analysis compares eigenvalues calculated from the actual data to eigenvalues generated on the basis of random data matrices matched to the actual data on sample size and number of factor analysis items (Patil et al., Reference Patil, Singh, Mishra and Donavan2008). The number of factors to extract is equal to the number of eigenvalues in the actual data that are greater than the corresponding eigenvalues in the random data.

A first EFA was then conducted using unweighted least squares with polychoric correlations, a method recommended when extracting factors from ordinal data (Basto and Pereira, Reference Basto and Pereira2012; Gaskin and Happell, Reference Gaskin and Happell2014). We selected promax oblique rotation given that oblique rotation methods allow the derived factors to be correlated (Costello and Osborne, Reference Costello and Osborne2005), consistent with our assumption of correlations between factors in the present study.

We compared several factor solutions to identify the ‘cleanest factor structure’ in terms of data and theoretical fit (Costello and Osborne, Reference Costello and Osborne2005: 3). Items were initially assessed for retention on the basis of the 0.40–0.30–0.20 rule: primary factor loadings ⩾0.40, secondary factor loadings <0.30, and ⩾0.20 difference between primary and secondary loadings (Howard, Reference Howard2016). Any remaining items with low communalities between 0.20 and 0.40 (Gaskin and Happell, Reference Gaskin and Happell2014) indicated that the factor solution accounted for a low proportion of variance in these items (Tabachnick and Fidell, Reference Tabachnick and Fidell2013). These items were evaluated for their theoretical contribution to the overall model, and considered for removal (Costello and Osborne, Reference Costello and Osborne2005). Furthermore, factors that consisted of only two items were considered for removal on the basis of their correlation with each other as well as with other items in the factor model (Tabachnick and Fidell, Reference Tabachnick and Fidell2013).

A second EFA was conducted with the items remaining after the first EFA. We report the mean item scores and standardized Cronbach’s α coefficients for the final PHCE subscales and overall scale, using a cut-off point of 0.70 as evidence of adequate internal consistency reliability (Nunnally and Bernstein, Reference Nunnally and Bernstein1994).

Results

Sample characteristics

As shown in Table 1, the majority of included participants were female, RNs, staff nurses, had attained a diploma as their highest level of education, were primarily employed outside of community-based health care settings, and were employed in communities with less than 10 000 population. Included participants were 47.0 years of age on average (SD=11.6) and had been registered to practice in Canada an average of 21.0 years (SD=13.0).

Exploratory factor analysis

Based on principal component analysis with data from 1587 participants, the KMO measure (0.91) and significant Bartlett’s test of sphericity [χ 2 (780)=23 859.80, P=0.000] pointed to a strong association among the scale items, which supported the use of factor analysis. Parallel analysis suggested an eight-factor solution, with eigenvalues generated by principal component analysis exceeding random values (in brackets) for the first eight factors: 9.68 (1.34), 2.65 (1.31), 2.27 (1.27), 1.60 (1.25), 1.49 (1.23), 1.47 (1.21), 1.38 (1.19), 1.21 (1.17), 1.10 (1.16).

In the first EFA (n=1587), we produced solutions of 5, 6, 7, 8, 9, and 10 factors for comparison. An eight-factor solution was optimal in terms of data fit as well as conceptual fit with the majority of the original subscales in the 40-item PHCE scale (Table 2). From the original 40 items, a total of 12 items were removed based on the 0.40–0.30–0.20 rule (Howard, Reference Howard2016). First, 11 items with primary factor loadings <0.40 were removed, as well as one item (C3) with a low factor loading of 0.41 and a low communality of 0.19 (Table 3). Second, the secondary factor loadings of all 40 items were <0.30 and therefore no items were removed on the basis of this criterion. Third, of the 12 items removed, 10 also demonstrated a difference of <0.20 between their primary and secondary factor loadings.

Table 2 Factor loadings of items included in the eight-factor solution based on first exploratory factor analysis

Bold font indicates primary factor loadings.

Table 3 Factor loadings of items removed from the 8-factor solution based on first exploratory factor analysis

Bold font indicates primary factor loadings.

With respect to the remaining 28 items with factor loadings ranging from 0.42 to 0.89, the majority demonstrated communalities in the recommended range (0.42–0.79). However, five items had communalities below 0.40, namely G1 and G4 (factor 3), F1 and F4 (factor 4), and C4 (factor 8). We made the decision to retain these five items given their theoretical contributions to the final scale.

The eight-factor solution (Table 2) included five factors (1, 2, 3, 5, and 7) that retained all of the items from five original dimensions (community participation, patient-centred care, intersectoral team, accessibility/availability, and interdisciplinary collaboration). Factor 4 included items that loaded from two separate original dimensions (population orientation and equity). Given that the population orientation items (D3 and D5) had higher loadings than the equity items (F1 and F4), and the equity items measured barriers between the community population and the participant’s workplace, we preserved the label ‘population orientation’ for Factor 4 and dropped the label ‘equity’. Factors 6 and 8 consisted of a reduced but sufficient number of items supporting two original dimensions (quality improvement and continuity) and were therefore retained. Although Factor 6 consisted of only two items, it was retained because these items were highly correlated with one another (>0.70) and uncorrelated or moderately correlated (<0.40) with other items. The factor loadings of items from the last original dimension (comprehensiveness) were <0.40, and as a result this factor was not included in the eight-factor solution. Therefore, the two original dimensions that were not included in the final eight-factor model were equity and comprehensiveness.

Based on the second EFA performed with the final 28 items, the total variance explained by the final eight-factor solution was 14.8%, with the proportion of variance explained by each factor 2.4% (community participation), 2.4% (patient-centred care), 2.1% (intersectoral team), 2.1% (accessibility/availability), 1.6% (interdisciplinary collaboration), 1.5% (quality improvement), 1.5% (population orientation), and 1.2% (continuity). The eight factors demonstrated moderate correlation (>0.30) between one another with the exception of: patient-centred care and quality improvement (0.27), continuity with all other factors except patient-centred care, and accessibility/availability with all other factors (Table 4).

Table 4 Factor correlation matrix of eight-factor solution based on second exploratory factor analysis

Reliability and summary statistics

Internal consistency reliability was estimated by the Cronbach’s α coefficient, which was 0.89 for the final 28-item PHCE Scale (Table 5). Six of the final eight subscales demonstrated α estimates ranging from 0.77 to 0.88, which exceeded the recommended standard of 0.70 (Nunnally and Bernstein, Reference Nunnally and Bernstein1994). The α estimates for two of the final eight subscales fell below 0.70, specifically continuity (α=0.61) and population orientation (α=0.61).

Table 5 Summary statistics of final 28-item Primary Health Care Engagement (PHCE) scale and subscales

Scale scoring: 1=strongly disagree; 2=disagree; 3=neutral; 4=agree; 5=strongly agree.

The mean item scores for the final 28-item PHCE Scale and eight subscales demonstrated low (3.1–4.0) to high agreement (4.1–5.0) among rural/remote nurses on workplace engagement in key dimensions of primary health care (Table 5). Mean scores ranged from a low of 3.1 (SD=0.7) on intersectoral team indicating a low level of agreement that collaboration exists between the primary workplace and community sectors, to a high of 4.2 (SD=0.6) on patient-centred care indicating high agreement that respect, safety, and confidentiality toward patients is present in the primary workplace. Overall, the mean PHCE Scale score of 3.6 (SD=0.5) indicated low agreement that the primary workplaces of rural/remote nurses were engaged in primary health care delivery.

Discussion

The present study describes an evaluation to provide validity evidence for an instrument to measure workplace engagement in primary health care delivery. Evidence for the construct validity of conclusions that may be drawn from an instrument is generally derived from five sources, including the scale’s content and internal structure (Cook and Beckman, Reference Cook and Beckman2006). In an earlier study, we provided content evidence to demonstrate that the PHCE Scale represents community-based/oriented primary health care (Kosteniuk et al., Reference Kosteniuk, Wilson, Penz, MacLeod, Stewart, Kulig, Karunanayake and Kilpatrick2016). Findings from the present study provide evidence of the internal structure of the PHCE Scale, based on an eight-factor structure identified in EFA and good internal consistency reliability (α=0.89).

Other provider-level instruments are available to assess health care sites on performance indicators of primary care (Schoen et al., Reference Schoen, Osborn and Huynh2006; Dahrouge et al., Reference Dahrouge, Hogg, Russell, Geneau, Kristjansson, Muldoon and Johnston2009) and PHC delivery (Johnston and Burge, Reference Johnston and Burge2013). However, the 28-item PHCE Scale differs from previous provider-level instruments in that it includes most of the principles common to contemporary community-based PHC approaches (eg, accessibility, community participation, and intersectoral collaboration) (Henderson et al., Reference Henderson, Koehne, Verrall, Gebbie and Fuller2014; White, Reference White2015; Baum et al., Reference Baum, Freeman, Sanders, Labonte, Lawless and Javanparast2016).

Based on the results of EFA, the original 40-item PHCE Scale across 10 subscales was reduced to 28 items across eight factors. Five of the final eight factors retained all of the items proposed in the original subscales (community participation, patient-centred care, intersectoral team, accessibility/availability, and interdisciplinary collaboration). The factor loadings of these items were well above the recommended cut-off, and reliability estimates of the five factors were satisfactory (α>0.70). Two of the final eight factors (continuity and quality improvement) lost items from the original subscales, however, the remaining items were sufficient in number and demonstrated adequate loadings for the factors to be retained. The last of the eight factors consisted of items loading from two separate original subscales (population orientation and equity), and therefore the label that was consistent with the highest loading items was retained (population orientation). Lower than expected reliability estimates (α<0.70) were demonstrated by two of the final factors (continuity and population orientation), which may be due to low correspondence between items or heterogeneity of concepts within the factors (Tavakol and Dennick, Reference Tavakol and Dennick2011).

The overall 28-item PHCE score provides a broad measure of workplace engagement in primary health care delivery, while the eight separate 2–4 item subscales allow measurement of particular aspects of PHC engagement. Higher scores on the PHCE Scale indicate a higher level of workplace participation in PHC delivery. The final 28-item scale directly represents three of four principles common to a contemporary approach to PHC (Henderson et al., Reference Henderson, Koehne, Verrall, Gebbie and Fuller2014; White, Reference White2015; Baum et al., Reference Baum, Freeman, Sanders, Labonte, Lawless and Javanparast2016) based on the original Alma Ata Declaration on Primary Health Care (World Health Organization, 1978): accessibility (accessibility/availability), community participation, and intersectoral collaboration (intersectoral team). The fourth element (equity) is indirectly represented in the population orientation subscale, indicating possible overlap in these two concepts or in the design of the items representing these two concepts. The remaining dimensions in the final PHCE Scale directly represent most elements of community-oriented (Levesque et al., Reference Levesque, Haggerty, Burge, Beaulieu, Gass, Pineault and Santor2011) and community-based PHC models (CIHR, 2015): a wide range of providers (interdisciplinary collaboration), person-centredness (patient-centred care), coordination and integration (continuity), population-centredness (population orientation), and quality improvement. An important element missing from the final PHCE Scale is comprehensiveness, the delivery ‘of a broad spectrum of health and social services’ (Levesque et al., Reference Levesque, Haggerty, Burge, Beaulieu, Gass, Pineault and Santor2011: 23) considered essential to community-based/oriented PHC. Although it is possible that as a conceptual category, comprehensiveness may not be an essential dimension of PHCE, it is more likely that the items did not adequately capture the complexity of this concept in the context of diverse rural/remote patient populations.

Some study limitations must be acknowledged. First, the proportion of participants that responded ‘not applicable’ was greater than 20% and as high as 49% for eight of the original 40 items across three subscales. We chose not to impute these ‘not applicable’ responses and other missing values, since including these values may have resulted in artificially high correlations (Tabachnick and Fidell, Reference Tabachnick and Fidell2013). Six of these items remained in the final 28-item scale (A2, A3, and A4 in accessibility/availability; G1, G2, and G3 in intersectoral team). The high proportion of ‘not applicable’ responses to the accessibility/availability items indicates that some participants may view questions regarding workplaces that ‘open’ and ‘close’ as irrelevant to their practice context. These items may be more relevant to community-based settings that offer regular office hours and appointment times than to institutional settings that offer 24/7 care. The high proportion of ‘not applicable’ responses to the intersectoral team items may reflect a ‘tyranny of the acute’ organizational culture that places a low emphasis on population health issues in some workplaces, as well as challenges to working intersectorally without formal supporting structures in place (CIHI, 2014). However, accessibility/availability and intersectoral collaboration are principles of a contemporary approach to PHC and therefore it is important and appropriate to retain these items in the final 28-item PHCE Scale.

A second limitation is that we applied a stringent listwise deletion criterion to exclude cases with at least one missing item, resulting in a large number of excluded cases (n=2235). Participants employed in community-based health care settings were more likely to provide a valid response to all of the scale items and to be included in the analysis than their counterparts in institutional settings (eg, hospital, long-term care). However, the majority of included participants were employed outside of community-based health care sites, indicating that the overall PHCE Scale may be applicable across a range of health care settings. We recommend the use of a case mean imputation strategy (El-Masri and Fox-Wasylyshyn, Reference El-Masri and Fox-Wasylyshyn2005) when employing the PHCE Scale for purposes other than psychometric testing, so as to obtain the maximum number of valid responses (see the PHCE Scale in the Appendix for specific imputation guidelines). Third, the study sample included only regulated nurses working in rural/remote Canadian communities, and as such this limits the generalizability of our findings in terms of professional discipline and geography. Future research should evaluate the instrument outside of these populations. Finally, the comprehensive 27-page RRNII survey provided an opportunity to administer the original 40-item PHCE Scale to regulated nurses working across rural/remote Canada. However, the purpose of the survey was to gather data on the overall nature of rural/remote nursing practice across five wide-ranging content domains such as work community and nursing practice (MacLeod et al., Reference MacLeod, Stewart, Kulig, Anguish, Andrews, Banner, Garraway, Hanlon, Karunanayake, Kilpatrick, Koren, Kosteniuk, Martin-Misener, Mix, Moffitt, Olynick, Penz, Sluggett, Van Pelt, Wilson and Zimmer2017) and as such we were unable to include other measures that may be expected to correlate with the PHCE Scale. Therefore, future research should focus on assessing the relations between the 28-item PHCE Scale and other variables, to provide further evidence of the construct validity of conclusions drawn from the scores (Cook and Beckman, Reference Cook and Beckman2006).

Conclusion

The most important finding of the present study is that the results supported an eight-factor structure of the PHCE Scale with 28 items in total, offering a comprehensive measure of workplace engagement in primary health care that is short and easy to administer. The scale is intended to be administered by researchers and health care planners to any health care provider likely to be involved in PHC delivery.

The 28-item PHCE Scale reflects the high priority placed on community participation in rural/remote PHC programs and services (Preston et al., Reference Preston, Waugh, Larkins and Taylor2010), and the importance of community participation in PHC more broadly as emphasized in recent position papers (Henderson et al., Reference Henderson, Koehne, Verrall, Gebbie and Fuller2014; Canadian Nurses Association, 2015; White, Reference White2015; Baum et al., Reference Baum, Freeman, Sanders, Labonte, Lawless and Javanparast2016) and PHC reform strategy reports (Alberta Health, 2014; Saskatchewan Ministry of Health, 2016). Although the scale may be more relevant to community-based than institutional health care settings such as hospitals and nursing homes, findings from the present study indicate that it is relevant to rural/remote providers from both settings. Further psychometric testing with a diversity of health care providers and settings is recommended.

Acknowledgements

The article stems from the study: ‘Nursing Practice in Rural and Remote Canada II’, led by Martha MacLeod, Norma Stewart and Judith Kulig (http:// ruralnursing.unbc.ca). The authors are grateful to the nurses who responded to this survey and to Nadine Mix, Leana Garraway, and Larine Sluggett for their assistance with data analysis.

Financial Support

The full study was funded by the Canadian Institutes of Health Research (MOP 130260).

Conflicts of Interest

None.

Ethical Standards

The ethics committees of the research team approved the RRNII survey: University of Northern British Columbia Research Ethics Board (E2013.0320.037.02), University of Saskatchewan (Behavioural Research Ethics Board Certificate of Approval), University of Lethbridge (Certificate of Human Participant research), Aurora College (Scientific Research License), University of Montreal (Hospital Maisonneuve-Rosemont), and Dalhousie University (Health Science Research Ethics Board letter of approval).

Appendix

Table A1 28-item Primary Heath Care Engagement (PHCE) Scale

Footnotes

Scoring: 1=strongly disagree; 2=disagree; 3=neutral; 4=agree; 5=strongly agree (including a ‘not applicable’ response option is not recommended).

Reverse items: 3, 18, 25, 26, 27, 28.

Subscales: quality improvement (1, 2); community participation (3, 4, 5, 6); patient-centred care (7, 8, 9, 10); accessibility-availability (11, 12, 13, 14); intersectoral team (15, 16, 17, 18); interdisciplinary collaboration (19, 20, 21); continuity (22, 23, 24); population orientation (25, 26, 27, 28).

Case mean imputation guidelines: in the four-item subscales, the case mean may be imputed where 25% or less of items is missing (ie, 1 item) (El-Masri and Fox-Wasylyshyn, Reference El-Masri and Fox-Wasylyshyn2005); if a participant’s subscale is missing two or more items, then that participant’s subscale should be discarded. Case mean imputation should not be performed in the two-item and three-item subscales; if a participant is missing one or more item, then that participant’s subscale should be discarded.

References

Alberta Health 2014: Alberta’s primary health care strategy. Retrieved 19 January 2017 from http://www.health.alberta.ca/documents/Primary-Health-Care-Strategy-2014.pdf.Google Scholar
Basto, M. and Pereira, J.-M. 2012: An SPSS R-menu for ordinal factor analysis. Journal of Statistical Software 46, 129.Google Scholar
Baum, F., Freeman, T., Sanders, D., Labonte, R., Lawless, A. and Javanparast, S. 2016: Comprehensive primary health care under neo-liberalism in Australia. Social Science and Medicine 168, 4352.CrossRefGoogle ScholarPubMed
Bloch, G., Rozmovits, L. and Giambrone, B. 2011: Barriers to primary care responsiveness to poverty as a risk factor for health. BMC Family Practice 12, 62.Google Scholar
Canadian Institute for Health Information (CIHI) 2006. Pan-Canadian Health Care Indicators. Volume 1 Ottawa: CIHI, 1109.Google Scholar
Canadian Institute for Health Information (CIHI) 2014: Exploring a population health approach in health system planning and decision-making. Retrieved 14 April 2017 from https://secure.cihi.ca/free_products/CIHI_Bridiging_Final_EN_web.pdf.Google Scholar
Canadian Institutes of Health Research (CIHR) 2015: Community-based primary health care. Retrieved 19 January 2017 from http://www.cihr-irsc.gc.ca/e/43626.html.Google Scholar
Canadian Nurses Association 2015: Primary health care (Position Statement). Retrieved from 19 January 2017 from https://cna-aiic.ca/~/media/cna/page-content/pdf-en/primary-health-care-position-statement.pdf?la=en.Google Scholar
Cook, D. and Beckman, T. 2006: Current concepts in validity and reliability for psychometric instruments: theory and application. The American Journal of Medicine 119, e7e16.Google Scholar
Costello, A. and Osborne, J. 2005: Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation 10, 19.Google Scholar
Dahrouge, S., Hogg, W., Russell, G., Geneau, R., Kristjansson, E., Muldoon, L. and Johnston, S. 2009: The comparison of models of primary care in Ontario (COMP-PC) study: methodology of a multi-faceted cross-sectional practice-based study. Open Medicine 3, 149164.Google Scholar
Davis, K., Schoen, C., Schoenbaum, S., Doty, M., Holmgren, A., Kriss, J. and Shea, K. 2007: Mirror, mirror on the wall: an international update on the comparative performance of American Health Care. The Commonwealth Fund publication number 1027. Retrieved 19 January 2017 from commonwealthfund.org.Google Scholar
Dillman, D., Smyth, J.D. and Christian, L.M. 2014. Internet, phone, mail, and mixed-mode surveys: the tailored design method, 4th edition. Hoboken, NJ: John Wiley & Sons, Inc.Google Scholar
du Plessis, V., Beshiri, R., Bollman, R.D. and Clemenson, H. 2001: Definitions of rural. Rural and Small Town Canada Analysis Bulletin. Retrieved 19 January 2017 from http://www.statcan.gc.ca/pub/21-006-x/21-006-x2001003-eng.pdf.Google Scholar
El-Masri, M.M. and Fox-Wasylyshyn, S.M. 2005: Missing data: an introductory conceptual overview for the novice researcher. Canadian Journal of Nursing Research 37, 156171.Google Scholar
Flocke, S. 1997: Measuring attributes of primary care: development of a new instrument. Journal of Family Practice 45, 6474.Google Scholar
Ford, D. 2016: Four persistent rural healthcare challenges. Healthcare Management Forum 29, 243246.Google Scholar
Galdikiene, N., Asikainen, P., Rostila, I., Geen, P., Balciunas, S. and Suominen, T. 2016: Organizational social context in primary health care. Nordic Journal of Nursing Research 26, 103111.Google Scholar
Gaskin, C. and Happell, B. 2014: On exploratory factor analysis: a review of recent evidence, an assessment of current practice, and recommendations for future use. International Journal of Nursing Studies 51, 511521.Google Scholar
Gibson, O., Lisy, K., Davy, C., Aromataris, E., Kite, E., Lockwood, C., Ritano, D., McBride, K. and Brown, A. 2015: Enablers and barriers to the implementation of primary health care interventions for Indigenous people with chronic diseases: a systematic review. Implementation Science 10, 71.Google Scholar
Henderson, J., Koehne, K., Verrall, C., Gebbie, K. and Fuller, J. 2014: How is primary health care conceptualized in nursing in Australia? A review of the literature. Health and Social Care in the Community 22, 337351.Google Scholar
Howard, M. 2016: A review of exploratory factor analysis decisions and overview of current practices: what are we doing and how can we improve? International Journal of Human-Computer Interaction 31, 5162.Google Scholar
Johnston, S. and Burge, F. 2013: Measuring provider experiences in primary health care: report on the development of a PHC Provider Survey for the Canadian Institute for Health Information. Ottawa: Bruyere Research Institute. Retrieved 19 January 2017 from http://www.cphcrin-rcrissp.ca.Google Scholar
Kellar, S. and Kelvin, E. 2013. Munro’s statistical methods for health care research, 6th edition. Philadelphia, PA: Wolters Kluwer, Lippincott Williams & Wilkins.Google Scholar
Kenny, A., Farmer, J., Dickson-Swift, V. and Hyett, N. 2015: Community participation for rural health: a review of challenges. Health Expectations 18, 19061917.Google Scholar
Kosteniuk, J., Wilson, E., Penz, K., MacLeod, M., Stewart, N., Kulig, J., Karunanayake, C. and Kilpatrick, K. 2016: Development and psychometric evaluation of the Primary Health Care Engagement (PHCE) Scale: a pilot survey of rural and remote nurses. Primary Health Care Research and Development 17, 7286.Google Scholar
Levesque, J.-F., Haggerty, J., Burge, F., Beaulieu, I., Gass, D., Pineault, R. and Santor, D. 2011: Canadian experts’ views on the importance of attributes within professional and community-oriented primary healthcare models. Healthcare Policy 7, 2130.Google Scholar
Liu, Q., Wang, B., Kong, Y. and Cheng, K.K. 2011: China’s primary health care reform. Lancet 377, 20642066.Google Scholar
MacLeod, M., Kulig, J., Stewart, N. and Pitblado, R. 2004: Nursing practice in rural and remote Canada: final report to Canadian Health Services Research Foundation. Retrieved 27 January 2017 from http://www.unbc.ca/sites/default/files/sections/rural-nursing/en/17-rrnfinalreport.pdf.Google Scholar
MacLeod, M., Stewart, N., Kulig, J., Anguish, P., Andrews, M.E., Banner, D., Garraway, L., Hanlon, N., Karunanayake, C., Kilpatrick, K., Koren, I., Kosteniuk, J., Martin-Misener, R., Mix, N., Moffitt, P., Olynick, J., Penz, K., Sluggett, L., Van Pelt, L., Wilson, E. and Zimmer, L. 2017: Nurses who work in rural and remote communities in Canada: a national survey. Human Resources for Health15, 34.Google Scholar
Nunnally, J. and Bernstein, I. 1994. Psychometric Theory, 3rd edition. New York: McGraw-Hill, Inc.Google Scholar
Olden, P. and Hoffman, K. 2011: Hospitals’ health promotion services in their communities: findings from a literature review. Health Care Management Review 36, 104113.Google Scholar
Patil, B., Singh, S., Mishra, S. and Donavan, D. 2008: Efficient theory development and factor retention criteria: abandon the ‘eigenvalue greater than one’ criterion. Journal of Business Research 61, 162170.Google Scholar
Preston, R., Waugh, H., Larkins, S. and Taylor, J. 2010: Community participation in rural primary health care: intervention or approach? Australian Journal of Primary Health 16, 416.Google Scholar
Saskatchewan Ministry of Health 2012: Saskatchewan Framework for Primary Health Care Report. Regina, Saskatchewan. Retrieved 19 January 2017 from www.health.gov.sk.ca/primary-health-care.Google Scholar
Saskatchewan Ministry of Health 2016: Saskatchewan Advisory Panel on health system structure report: optimizing and integrating patient-centred care. Retrieved 19 January 2017 from http://s3.documentcloud.org/documents/3251960/Saskatchewan-Advisory-Panel-on-Health-System.pdf.Google Scholar
Schoen, C., Osborn, R. and Huynh, P. 2006: On the front lines of care: primary care doctors’ office systems, experiences, and views in seven countries. Health Affairs 25, w555w571.Google Scholar
Shi, L., Starfield, B. and Xu, J. 2001: Validating the adult primary care assessment tool. Journal of Family Practice 50, 1171.Google Scholar
Strumpf, E., Levesque, J.-F., Coyle, N., Hutchinson, B., Barnes, M. and Wedel, R. 2012: Innovative and diverse strategies toward primary health care reform: lessons learned from the Canadian experience. Journal of the American Board of Family Medicine 25, S27S33.CrossRefGoogle ScholarPubMed
Tabachnick, B. and Fidell, L. 2013. Using multivariate statistics, 6th edition. Boston, MA: Pearson Education.Google Scholar
Tavakol, M. and Dennick, R. 2011: Making sense of Cronbach’s alpha. International Journal of Medical Education 2, 5355.Google Scholar
Thomas, S., Wakerman, J. and Humphreys, J. 2014: What core primary health care services should be available to Australians living in rural and remote communities? BMC Family Practice 15, 143.Google Scholar
Ward, B.M., Buykx, P.F., Tham, R., Kinsman, L. and Humphreys, J.S. 2014: Investing in longitudinal studies of primary healthcare: what can we learn about service performance, sustainability and quality? Rural and Remote Health 14, 3059.Google Scholar
White, F. 2015: Primary health care and public health: foundations of universal health systems. Medical Principles and Practice 24, 103116.Google Scholar
World Health Organization 1978: Declaration of Alma-Ata: Report of the International Conference on Primary Health Care Alma-Ata, USSR. Retrieved 11 July 2017 from http://apps.who.int/iris/bitstream/10665/39228/1/9241800011.pdf.Google Scholar
World Health Organization 1986: Ottawa charter for health promotion. Ottawa. Retrieved January 19, 2017 from http://www.phac-aspc.gc.ca/ph-sp/docs/charter-chartre/pdf/charter.pdf.Google Scholar
Wong, S., Browne, A., Varcoe, C., Lavoi, J., Smye, V., Godwin, O., Littlejohn, D. and Tu, D. 2011: Enhancing measurement of primary health care indicators using an equity lens: an ethnographic study. International Journal for Equity in Health 10, 38.Google Scholar
Figure 0

Table 1 Characteristics of Nursing Practice in Rural and Remote Canada II survey respondents included and excluded from analysis

Figure 1

Table 2 Factor loadings of items included in the eight-factor solution based on first exploratory factor analysis

Figure 2

Table 3 Factor loadings of items removed from the 8-factor solution based on first exploratory factor analysis

Figure 3

Table 4 Factor correlation matrix of eight-factor solution based on second exploratory factor analysis

Figure 4

Table 5 Summary statistics of final 28-item Primary Health Care Engagement (PHCE) scale and subscales

Figure 5

Table A1 28-item Primary Heath Care Engagement (PHCE) Scale