Skip to main content Accessibility help
×
Home

Information:

  • Access
  • Open access

Actions:

      • Send article to Kindle

        To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Promoting cross-jurisdictional primary health care research: developing a set of common indicators across 12 community-based primary health care teams in Canada
        Available formats
        ×

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Promoting cross-jurisdictional primary health care research: developing a set of common indicators across 12 community-based primary health care teams in Canada
        Available formats
        ×

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Promoting cross-jurisdictional primary health care research: developing a set of common indicators across 12 community-based primary health care teams in Canada
        Available formats
        ×
Export citation

Abstract

Aim

To describe the process by which the 12 community-based primary health care (CBPHC) research teams worked together and fostered cross-jurisdictional collaboration, including collection of common indicators with the goal of using the same measures and data sources.

Background

A pan-Canadian mechanism for common measurement of the impact of primary care innovations across Canada is lacking. The Canadian Institutes for Health Research and its partners funded 12 teams to conduct research and collaborate on development of a set of commonly collected indicators.

Methods

A working group representing the 12 teams was established. They undertook an iterative process to consider existing primary care indicators identified from the literature and by stakeholders. Indicators were agreed upon with the intention of addressing three objectives across the 12 teams: (1) describing the impact of improving access to CBPHC; (2) examining the impact of alternative models of chronic disease prevention and management in CBPHC; and (3) describing the structures and context that influence the implementation, delivery, cost, and potential for scale-up of CBPHC innovations.

Findings

Nineteen common indicators within the core dimensions of primary care were identified: access, comprehensiveness, coordination, effectiveness, and equity. We also agreed to collect data on health care costs and utilization within each team. Data sources include surveys, health administrative data, interviews, focus groups, and case studies. Collaboration across these teams sets the foundation for a unique opportunity for new knowledge generation, over and above any knowledge developed by any one team. Keys to success are each team’s willingness to engage and commitment to working across teams, funding to support this collaboration, and distributed leadership across the working group. Reaching consensus on collection of common indicators is challenging but achievable.

Footnotes

a

Centre intégré universitaire de santé et de services sociaux du Saguenay-Lac-Saint-Jean, Chicoutimi, Québec, Canada.

Background

Strong primary care systems are associated with better patient outcomes, particularly for those with chronic conditions (Hansen et al., 2015). Yet around the globe, primary care clinicians report challenges in coordinating care and delivering care to the most complex and vulnerable patients (Osborn et al., 2015). This points to the need for targeted efforts to innovate in the delivery of primary health care with a particular focus on strategies to effectively reach the most vulnerable patients.

Research in primary care plays a critical role in informing, evaluating, and helping improve the delivery and organization of health care services (Hutchison et al., 2011; Hutchison and Glazier, 2013). The diversity of primary care activity within and across jurisdictions (eg, provinces, states, countries) presents an opportunity for learning about which innovations could be spread and scaled-up to provide better service to vulnerable populations and patients with chronic diseases. By comparing changes in primary care delivery across geographies and provincial/territorial boundaries, researchers, together with clinicians, decision-makers, and patients, can produce knowledge about what is needed to spread promising innovations across regions and which are reproducible on a larger scale.

A pan-Canadian mechanism for common measurement of the impact of primary care innovations across Canada is lacking. Currently, pan-Canadian measurement and reporting has been more focused on the acute care sector. For example, a report released on the performance of primary care in Canada in 2016 reported on only 16 of the 51 indicators that policy makers and clinicians had deemed important in 2012 [Canadian Institute for Health Information (CIHI), 2016]. This suggests that little progress has been made toward collecting and reporting primary care indicators; further emphasizing the need for developing common and shared data sources in primary care. More recently Canadian data using the Quality and Costs of Primary Care (QUALICO-PC) indicators have been reported from Canada (Wong et al., 2015; Breton et al., 2016; Anisimowicz et al., 2017; Katz et al., 2017).

In order to speed up progress in collecting data using the same measures and data sources across research projects and facilitating knowledge exchange across jurisdictions, the Canadian Institutes of Health Research (CIHR, 2013) and its provincial funding partners required cross-jurisdictional collaboration on indicator measurement and evaluation as part of a signature research initiative on community-based primary health care (CBPHC). The first of its kind in funding structure, the initiative invested $33.4 million in funding to the 12 multi-jurisdictional five-year team grants. These 12 teams are undertaking programmatic, cross-jurisdictional, and interdisciplinary research to develop, implement, evaluate, and compare innovations in primary care. In order to facilitate collaboration across the 12 teams, a condition of this funding was that a portion of the grant money could only be spent on collaboration and cross-team data collection and analysis of common indicators. We report here on the process by which the 12 teams worked together and fostered cross-jurisdictional collaboration, including collection of common indicators with the goal of using the same measures and data sources.

Methods

Initial conditions of funding

Twelve CBPHC team grants were selected by a rigorous international peer review process and funded by CIHR (2012) and partners in 2013. The signature initiative’s key research areas included: improving access for vulnerable populations and innovations in chronic disease management and prevention. To encourage building capacity and scaling-up successful innovations, each team was required to include at least two decision-makers from different jurisdictions, at least one health professional, and a community-engagement process to provide feedback on the proposed care delivery model (CIHR, 2012).

Each team was required to allocate $50 000 CAD/year over the course of the grant (total per team=$250 000) for reviewing, agreeing, and operationalizing common measurement and assessment strategies. Although cross-collaboration was among the requirements of receiving the funding, there were no a priori decisions on how the teams would work together.

Working group formation

A working group was established with representatives from each of the 12 teams as well as the funder during the first year of funding. The primary objective of this group was to develop a common set of indicators to describe the knowledge generated by these 12 cross-jurisdictional teams and to develop a process to work across the teams (Riddell et al., 2017). The group met through a series of teleconferences and face-to-face meetings, and was chaired by one of the 12 teams’ principal investigators.

Overarching research questions

The working group agreed upon collecting indicators within several core dimensions of primary care with the intention of addressing three objectives across the 12 teams:

  • Describe the impact of improving access to CBPHC, particularly for vulnerable populations;

  • Examine the impact of alternative models of chronic disease prevention and management in CBPHC on patient and system outcomes; and

  • Describe the structures and context that influence the implementation, delivery, cost, and potential for scale-up of CBPHC innovations.

Procedures for consensus on indicators and measures

The working group undertook an iterative process of discussion, consideration of existing primary care measures from the literature (Starfield, 2001; Hogg et al., 2008; Watson et al., 2009) and stakeholder engagement. We incorporated the CIHI (2012) pan-Canadian primary health care indicators and previous work developed in the Canadian context from measurement experts in primary care (Fortin et al., 2012; Haggerty et al., 2012; Wong et al., 2015). Teams were expected to incorporate the agreed upon common measures into their respective programs and to report on them.

In choosing measures and indicators, specific efforts focused on refining them based on feedback from teams’ First Nation and Inuit partners. Team members were asked to reach common ground about the priority dimensions to be included in data collection, where possible, across all of the 12 teams.

Data collection on their use of the common indicators

Teams were surveyed using self-administered questionnaires on their planned research programs, including study design, methods of data collection, data sources, and study populations, as well as their coverage of the common indicators, and whether they were collecting the exact or modified versions of the indicators.

Findings

Description of the teams, their research, and sources of data

The teams were heterogeneous in relation to their specific research questions, structure, and methodology, each with several studies or sub-studies using a variety of methods (Table 1). The teams’ projects were similar in the sense that they were written for internal validity and intervention fidelity with potential for generalizability of results. The teams employed a range of study designs including case studies, cross-sectional observational studies, pre–post intervention, randomized controlled trials, and longitudinal study designs. Taken together, the teams covered all of Canada with the exception of one province and one territory (Saskatchewan, Yukon, respectively) (Table 1). Three teams have Indigenous partners and three are collaborating internationally. Nine of the 12 teams are collecting data in English and French. Some teams are collecting data in other languages, including Cantonese, Greek, and Italian.

Table 1 Location and brief description of 12 teams projects

NL=Newfoundland; NB=New Brunswick; NS=Nova Scotia; PE=Prince Edward Island; ON=Ontario; AB=Alberta; BC=British Columbia; MB=Manitoba; QC=Quebec; PHC=primary health care; NU=Nunavut; NW=Northwest Territories.

More information about the projects can be found at http://www.cihr-irsc.gc.ca/e/45817.html

The dimensions of primary care and the agreed upon common indicators

Based on the funded teams’ proposals and contexts, feasibility, collective expertise, and primary health care indicators developed by the CIHI (2012), the working group agreed upon five primary health care dimensions, namely, access, comprehensiveness, coordination, effectiveness, and equity (see Table 2).

Table 2 Coverage of agreed upon primary care indicators across the 12 teams

PHC=primary health care; Pa=patient; Pr=clinician; Org=organization; HAD=health administrative data.

We identified 19 indicators covering the five dimensions that could be measured using publicly available items and scales (see Additional File 1). Common indicators in addition to those within the five dimensions include population/patient characteristics, multimorbidity, patient-reported outcome measures (PROMs) of health and well-being [VR-12-SF (Ware et al., 1996) and EQ-5D-5L (Xie et al., 2016)], health care utilization, and health care costs. All scales, including the Team Climate Inventory (Anderson and West, 1998) were selected based on having appropriate reliability and validity, prior use in primary care, user burden (where possible, items that were already being collected by some teams or could be easily added to planned data collection) and cost (with a preference for tools that are available free of charge to facilitate sustainability of this work) (Figure 1).

Figure 1 The process and criteria of development of the common indicators among the 12 teams

Data sources included patient, provider, and organizational surveys, and common algorithms in health administrative data. Validated tools for use with administrative health data include common algorithms for identifying age strata or examining multimorbidity, as examples.

Methods used to collect the data included interviews or secondary analysis of health administrative data after an approved data access request was completed (five of these teams are linking these data with other sources such as survey data) (Table 3).

Table 3 Sources and types of data collection used by the 12 teams

a Other data collected by teams include items such as community readiness surveys, volunteer satisfaction surveys, and publically available statistical data.

Coverage of indicators by the teams and sources of data

Some teams had started collecting data prior to the funding decision and formation of the working group. Where possible, the agreed upon measures were incorporated into teams’ research plans after the funding decisions were made and the working group was formed. Some teams are collecting modified versions of the agreed measures.

The most commonly collected indicators are in the dimensions of access (all teams), coordination (11/12 teams), and equity (11/12), whereas the least commonly collected indicators are related to health care costs (six teams) (Table 2). In addition, all teams are collecting measures related to patient characteristics, multimorbidity, and health care utilization.

Group process for cross-team collaboration

Following collaboration on the common indicators, the working group devised an authorship policy ratified by the principal investigators of all teams. This policy is informed by previous policies made by some of the 12 teams and their institutes as well as academic publishing guidelines [Hogg et al., 2014; International Committee of Medical Journal Editors (ICMJE), 2017].

The working group is now developing and implementing cross-team collaborative projects using data collected through the common indicators. Cross-team project ideas are pitched to the group in the form of one-pager summaries and smaller working groups are formed to lead the studies. Progress is shared with the larger group and other researchers can join at different stages. Cross-team projects are facilitated with the help of a cross-team research coordinator.

This work provides the largest pan-Canadian opportunity for cross-jurisdictional learning and knowledge generation on the processes, outputs, and patient-reported outcomes in primary care service delivery. For example, across the teams, we now have the largest source of data on team functioning in primary care practices and the largest source of data on PROMs in the Canadian primary care context.

Discussion

This unprecedented process of collaboration across 12 interdisciplinary CBPHC research teams operating across Canada in 13 jurisdictions is a unique opportunity for new knowledge generation that exceeds beyond the knowledge outputs of any one team. We have successfully built the infrastructure and fostered partnerships across the teams to advance primary care research and innovation. Across the teams, there are over 120 individual researchers, staff, and students who have developed ways for working together across Canada and now share common ways of measuring key dimensions of primary care.

The collected common indicators can serve as a standard set of items and scales for common primary care performance reporting across Canada; they could complement the gaps in data for existing reporting initiatives such as reporting on the CIHI (2016) pan-Canadian primary care indicator set. As an added value to such quantitative measures, our combined efforts provide a richness of multiple data sources, diverse perspectives, and contextual factors. Finally, cross-comparative analysis of data collected through validated tools such as EQ-5D-5L and Team Climate Inventory in different contexts could inform of their utility in the Canadian primary care.

In the context of research impact (Greenhalgh et al., 2016), the development of common metrics, a data infrastructure, and a collaborative process across the 12 teams has the potential to make a dramatic contribution to the key aims of the signature initiative. In addition, because of the heterogeneity of the teams and the diversity of local context, the learnings of the 12 teams and the working group bring more real world relevance to measurement of primary care. The data collected through the common indicators, complemented with local context, can answer questions such as ‘what works for whom in what circumstances’ (Pawson and Tilley, 2001). The working group across the teams is an innovation in how to best share and reflect on heterogeneous and common data to foster improvements in primary care delivery and organization. Moreover, we have developed capacity across these teams to collect data for future analysis and bring understanding of the context about outcomes related to spread and scale of innovations in delivery and organization of primary care.

A key factor in the success of the common indicators working group is each team’s willingness to collaborate openly, engage with each other and commitment to the process. Other key success factors include leadership of the working group that entailed active engagement of all members, a research coordinator, and regular liaising with the funder and different teams based on their needs. An additional condition needed for this work is embedded funding. By allocating funding and mandating cross-team cooperation, CIHR and its partners has created an environment for collaboration rather than competition.

This work began with broad aims and varied methodologies, which required dedicated collaboration on behalf of the research teams in order to establish common ground and move towards the cross-team objectives (CIHR, 2012). The lack of standardization in applying the measures and data collection will limit the kinds of analyses possible. Future analyses will require a thorough assessment of potential bias introduced by the heterogeneity of data collection methods. At the very least, we now have a foundation of researchers and trainees from which to build cross-team research and can incorporate important learnings from other examples of cross-team research in primary care, such as the Agency for Health care Research and Quality Patient-Centred Medical Home research wherein specific goals and objectives were defined at the beginning and projects were carried out independently, with outcomes pooled at the end (Genevro and Meyers, 2013).

Reaching consensus on common indicators was challenging at times. In part, this was a normal process of group formation where we moved through the norming, storming, and forming stages (Tuckman, 1965). Moreover, there has been turnover of investigators and staff in this group. Much of our disagreement was influenced by the high value placed on traditional approaches to research methods. These methods are guided by fidelity to a priori specification of hypotheses-testing, which drives sampling, randomization, and data collection processes.

An ongoing challenge is having the working group members act as liaisons between the working group and their individual teams, the latter of which include community partners and other diverse stakeholder groups. Finding ways to optimize the group process and utilize all expertise (funders, researchers, and community partners) to develop a mutually acceptable set of measures and common approach to data collection are resource and time intensive. Indeed, each team found it important to respect the values and preferences of their represented sub-populations while also collaborating with other teams and moving toward common goals.

Other challenges associated with conducting cross-team research include confidentiality related to data sharing and ensuring accurate and meaningful comparisons of results across teams with divergent study designs, target patient populations and that not all teams were using methods compatible with the entire set of measures. Ownership of data is of critical importance in relation to working with First Nations and Inuit partners; teams who were specifically committed to participatory research methods and principles of Ownership, Control, Access, Possession (First Nations Information Governance Centre, 2014) meant that decisions to collect common indicators and participate in cross-team analysis of data require additional memoranda of understanding with Indigenous community partners. In that regard, the data-sharing processes opted by the common indicators working group are experiments that can be replicated in other contexts to further facilitate cross-jurisdictional and collaborative PHC research.

Conclusions

Twelve CBPHC research teams, supported by specific funding, formed a working group and engaged in a process to agree on and collect common data indicators for the generation of knowledge beyond any one team’s ability. As a result of this signature initiative, a wealth of data stemming from the common indicators has been generated in primary health care. Collected process, outcome, and contextual data from the 12 teams will be analyzed using mixed methods to describe elements of the interventions and the contexts that will influence their implementation and scale-up. Future work should define measures a priori within request for proposals and require that teams build this into their applications. Such an approach could enable funders and researchers to synthesize findings across funded studies to be able to give a cohesive picture of findings from an initiative.

The 12 teams collaboration has laid the groundwork for measurement in future CBPHC endeavors, and while the measures may not be directly transferable to other high-priority areas of health care (eg, end-of-life care) our framework and processes can be used to inspire and fast-track similar future initiatives. Our findings can provide guidance for those at the coalface of innovation in primary care from practice-level service delivery and planning through to national performance measurement and reporting.

Acknowledgments

The authors are thankful to Kue Young, Stewart Harris, Janusz Kaczorowski, Erica Di Ruggiero, Robyn Tamblyn, Nancy Edwards, Meg McMahon, Meredith Kratzmann, Bojana Petrovic, and Dawn Mooney for their comments and contributions.

Authors’ Contribution

S.W. and J.L. wrote the initial draft of the manuscript. All teams participated in the process of developing the common indicators. All authors contributed to the interpretation, writing, and approval of the final manuscript.

Financial Support

This study has been supported by CBPHC signature initiative grants from the CIHR. The funder did not play a role in study design, data collection, analysis, and interpretations of the data.

Conflicts of Interest

None.

References

Anderson, NR West, MA (1998) Measuring climate for work group innovation: development and validation of the team. Journal of Organizational Behavior 19, 235258.
Anisimowicz, Y, Bowes, AE, Thompson, AE, Miedema, B, Hogg, WE, Wong, ST, Katz, A, Burge, F, Aubrey-Bassler, K, Yellanc, GS Wochis, WP (2017) Computer use in primary care practices in Canada. Canadian Family Physician Medecin de Famille Canadien 63, e284e290.
Canadian Institute for Health Information (CIHI) (2012) Pan-Canadian primary health care indicator update report. Retrieved 11 August 2014 from https://secure.cihi.ca/estore/productFamily.htm?locale=en&pf=PFC2000&lang=en.
Canadian Institute for Health Information (CIHI) (2016) Primary health care in Canada: a chartbook of selected indicator results, 2016. Ottawa, Ontario: CIHI.
Canadian Institutes of Health Research (CIHR) (2012) Team grant: community-based primary healthcare (the transformative community-based primary healthcare initiative). Retrieved 14 February 2017 from https://www.researchnet-recherchenet.ca/rnr16/viewOpportunityDetails.do?progCd=10282&language=E&fodAgency=CIHR&view=browseArchive&browseArc=true&org=CIHR.
Canadian Institutes of Health Research (CIHR) (2013) Community-based primary health care featured research. Retrieved 27 April 2017 from http://www.cihr-irsc.gc.ca/e/45817.html.
First Nations Information Governance Centre (2014) Ownership, control, access and possession (OCAP TM): the path to First Nations information governance. First Nations Information Governance Centre.
Fortin, M, Stewart, M, Poitras, ME, Almirall, J Maddocks, H (2012) A systematic review of prevalence studies on multimorbidity: toward a more uniform methodology. Annals of Family Medicine 10, 142151.
Genevro, JL Meyers, DS (2013) Lessons learned from the study of primary care transformation. Annals of Family Medicine 11, 5155.
Greenhalgh, T, Raftery, J, Hanney, S Glover, M (2016) Research impact: a narrative review. BMC Medicine 14, 78.
Haggerty, JL, Roberge, D, Freeman, GK, Beaulieu, C Breton, M (2012) Validation of a generic measure of continuity of care: when patients encounter several clinicians. Annals of Family Medicine 10, 443450.
Hansen, J, Groenewegen, PP, Boerma, WG Kringos, DS (2015) Living in a country with a strong primary care system is beneficial to people with chronic conditions. Health Affairs 34, 15311537.
Hogg, W, Rowan, M, Russell, G, Geneau, R Muldoon, L (2008) Framework for primary care organizations: the importance of a structural domain. International Journal for Quality in Health Care 20, 308313.
Hogg, W, Donskov, M, Russell, G, Liddy, C, Johnston, S Mayo-Bruinsma, L (2014) Approach to publishing for large health services research projects. Canadian Family Physician. Retrieved 11 May 2018 from http://www.cfp.ca/content/cfp/60/9/854.full.pdf.
Hutchison, B, Levesque, JF, Strumpf, E Coyle, N (2011) Primary health care in Canada: systems in motion. The Milbank Quarterly 89, 256288.
Hutchison, B Glazier, R (2013) Ontario’s primary care reforms have transformed the local care landscape, but a plan is needed for ongoing improvement. Health Affairs 32, 695703.
International Committee of Medical Journal Editors (ICMJE) (2017) Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. Retrieved 11 May 2018 from http://www.icmje.org/icmje-recommendations.pdf.
Katz, A, Herpai, N, Smith, G, Aubrey-Bassler, K, Breton, M, Boivin, A, Hogg, W, Miedema, B, Pang, J, Wodchis, WP Wong, ST (2017) Alignment of Canadian primary care with the patient medical home model: a QUALICO-PC study. Annals of Family Medicine 15, 230236.
Miedema, B, Easley, J, Thompson, AE, Boivin, A, Aubrey-Bassler, K, Katz, A, Hogg, WE, Breton, M, Francoeur, D, Wong, ST Wodchis, WP (2016) Do new and traditional models of primary care differ with regard to access? Recherche Les modèles de pratique nouveaux et traditionnels diffèrent-ils en termes d’ accessibilité aux soins de santé primaires? Canadian Family Physician 62, 5461.
Osborn, R, Moulds, D, Schneider, EC, Doty, MM, Squires, D Sarnak, DO (2015) Primary care physicians in ten countries report challenges caring for patients with complex health needs. Health Affairs 34, 21042112.
Pawson, R Tilley, N (2001) Realistic evaluation bloodlines. American Journal of Evaluation 22, 317324.
Riddell, MA, Edwards, N, Thompson, SR, Bernabe-Ortiz, A, Praveen, D, Johnson, C, Kengne, AP, Liu, P, McCready, T, Ng, E, Nieuwlaat, R, Ovbiagele, B, Owolabi, M, Peiris, D, Thrift, AG, Tobe, S, Yusoff, K GACD Hypertension Research Programme (2017) Developing consensus measures for global programs: lessons from the Global Alliance for Chronic Diseases Hypertension research program. Globalization and Health 13, 17.
Starfield, B (2001) Primary care: balancing health needs, services and technology. International Journal of Integrated Care 1, 15684156.
Tuckman, B (1965) Developmental sequence in small groups: Tuckman’s forming storming norming performing overview. Psychological Bulletin 63, 384399.
Ware, JJ, Kosinski, MM Keller, SSD (1996) A 12-Item Short-Form Health Survey: construction of scales and preliminary tests of reliability and validity. Medical Care 34, 220233.
Watson, DE, Broemeling, AM Wong, ST (2009) A results-based logic model for primary healthcare: a conceptual foundation for population-based information systems. Healthcare Policy 5 (Special Issue), 33.
Wong, ST, Chau, LW, Hogg, W, Teare, GF, Miedema, B, Breton, M, Aubrey-Bassler, K, Katz, A, Burge, F, Boivin, A, Cooke, T, Francoeur, D Wodchis, WP (2015) An international cross-sectional survey on the Quality and Costs of Primary Care (QUALICO-PC): recruitment and data collection of places delivering primary care across Canada. BMC Family Practice 16, 20.
Xie, F, Pullenayegum, E, Gaebel, K, Bansback, N, Bryan, S, Ohinmaa, A, Poissant, L, Johnson, JA Canadian EQ-5D-5L Valuation Study Group (2016) A time trade-off-derived value set of the EQ-5D-5L for Canada. Medical Care 54, 98105.