Skip to main content Accessibility help
×
Home

Information:

  • Access
  • Open access

Actions:

      • Send article to Kindle

        To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Participatory methods to support team science development for predictive analytics in health
        Available formats
        ×

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Participatory methods to support team science development for predictive analytics in health
        Available formats
        ×

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Participatory methods to support team science development for predictive analytics in health
        Available formats
        ×
Export citation

Abstract

Predictive analytics in health is a complex, transdisciplinary field requiring collaboration across diverse scientific and stakeholder groups. Pilot implementation of participatory research to foster team science in predictive analytics through a partnered-symposium and funding competition. In total, 85 stakeholders were engaged across diverse translational domains, with a significant increase in perceived importance of early inclusion of patients and communities in research. Participatory research approaches may be an effective model for engaging broad stakeholders in predictive analytics.

Introduction

Predictive analytics in health is an emerging transdisciplinary field utilizing techniques from computer science (e.g., machine learning, signal processing), statistics, clinical medicine, and social and behavioral sciences to predict individual and group-level health outcomes [1, 2]. It involves the collection, merging, and analysis of multiple types of individual-level data such as electronic health records, publicly-available administrative data, mobile phone activity, and increasingly, passive sensing technologies [3]. With these advances, concerns have emerged regarding data privacy, ownership, and the risk/benefits related to predicting sensitive, individual-level health outcomes and behaviors [4]. This is especially relevant in low-income, racial/ethnic minority populations that often have limited trust in and access to research opportunities [5, 6]. Team science approaches prioritizing broad stakeholder engagement in research leadership may facilitate advances in predictive analytics by integrating knowledge and priorities across disciplines and perspectives.

As noted by the Institute of Medicine [7] and the National Institutes of Health [8], team science is a collaborative, transdisciplinary approach recommended for accelerating development of clinical translational science into public health impact. Team science addresses complex, multi-faceted questions by including scientists from diverse disciplines, and in some models, the end-users of biomedical research (e.g., community members, patients, clinicians, healthcare systems, payers). To foster effective team science, the Institute of Medicine report included several key recommendations including supporting high diversity of membership, shared vocabularies, and deep knowledge integration. Although previous reports have described factors associated with effective translational team science broadly [9, 10], little is known about how to encourage team science in the emerging field of predictive analytics in health. Community-partnered participatory research (CPPR), a variant of community-based participatory research [11], was developed to support effective partnerships between researchers and stakeholders from under-resourced communities, may be one approach to foster team science in this field [6]. CPPR focuses on early engagement of stakeholders in research co-leadership. The goal is to build trust, engagement and a shared culture across stakeholders through equal power sharing, identification of common interests, and ongoing workgroups [12]. These participatory techniques have been used successfully in bridging cultural gaps to form partnerships between stakeholders from under-resourced communities and academic institutions [6, 11], with increasing examples applied to health informatics [13]. However, it is unclear how these methods may help foster partnerships across the disparate academic disciplines, translational domains, and institutions relevant to predictive analytics.

In this report, we pilot a Clinical Translational Science Award (CTSA)-funded engagement approach to support predictive analytic research that both implements partnered research methods in organizing a team science symposium and introduces these principles to participants through group learning. We primarily focused on addressing the challenge of building partnerships between stakeholders across diverse academic disciplines, translational domains and institutions. We hypothesized that this would support engagement of translational researchers in predictive analytics across disciplines and increase participants’ perceived importance of stakeholder inclusion in research co-leadership.

Methods

Participatory Engagement

We implemented participatory research (CPPR) best practices to stimulate transdisciplinary planning for a scientific symposium on translational team science in predictive health analytics. This included early and diverse stakeholder inclusion, shared decision-making, and building capacity for partnered planning and implementation of research-informed programs [7, 13]. This scientific development initiative and evaluation were funded through a University of California Los Angeles (UCLA) Clinical and Translational Science Institute (CTSI) Catalyst Award. We applied partnered techniques to engage diverse academic disciplines and institutions in planning and participating in a symposium and pilot award program. All planning was conducted through a stakeholder advisory panel with members from 3 CTSA institutions including University of California Los Angeles, University of Southern California, and University of Pittsburgh. The UCLA CTSI provided pilot funding for this project. Stakeholders represented diverse disciplines relevant to predictive health analytics (e.g., statistics, informatics, computer science, clinical medicine, health services research, social sciences, healthcare providers, and administrators).

Symposium and Pilot Competition

The partnered approach to planning through the stakeholder advisory board over a 2-month period resulted in development of 2 structured activities: (1) a day-long symposium and (2) a pilot competition, to encourage new transdisciplinary, translational team science around predictive analytics in health. The symposium and pilot competition were held in December 2013. Additionally, 3-year outcomes for the pilot competitions were obtained in December 2016. During pre-symposium planning, the advisory board identified three themes for the symposium including opportunities for utilizing health system data, predictive analytic methods, and partnered engagement strategies. Three keynote talks were organized to address these themes: (1) reduction of hospital readmission after acute cardiac events using predictive analytics of electronic health record services data, (2) engineering approaches to behavioral signal processing including use of mobile sensors, and (3) an overview of CPPR and implications for predictive analytics.

Advisory board members identified individuals and relevant email distribution lists (e.g., departmental or interest group) to send invitations for prospective attendees. The symposium consisted of an in-person meeting in Long Beach, CA with simultaneous, online participation from Pittsburgh, PA, San Francisco, CA and a keynote address from Dallas, TX. To encourage participation and co-learning, the symposium started with an opportunity for in-person participants to present one slide on their work or interests and remote attendees to introduce their work via videoconference. Thirty-seven of the in-person attendees presented a slide (52%) and spoke approximately 1–3 minutes each. Examples include a summary of interests (e.g., analytic, methodological, or content expertise), a key figure or table from recent studies, and conceptual overviews. There were 2 breakout sessions to discuss key questions and topics addressed in the keynote talks. The symposium concluded with a discussion of the pilot competition for 2, $15,000 awards open to teams emerging from the symposium. Pilot awards were prioritized for teams with researchers forming collaborations across disciplines that they had not worked in before and teams with patients, families, and/or community leaders.

Participant Survey

Evaluation surveys (pre-symposium and post-symposium) were distributed as a single packet to attendees at registration with a request to complete the post-symposium survey at the end of the symposium and return to staff before leaving. Pre-symposium surveys consisted of 15 items including organizational affiliations, education, techniques directly used by participants or through collaborations, challenges to work and collaboration, translational domains of their work (e.g., T0–T4) [14], and perspectives on community-partnered methods. Post-symposium surveys consisted of 7 items assessing perspectives related to community-partnered methods and interest in collaborating across disciplines. Of in-person attendees, 63% (n=44/70) completed the pre-symposium survey and 59% (n=40/70) the post-symposium survey. Survey consent was oral and no incentives were offered for participation. We report results of paired, one-tailed t-tests calculated to assess differences between matched survey items, pre-symposium and post-symposium. A p-value of <0.05 was considered statistically significant. The UCLA Institutional Review Board approved of all methods.

Results

The partnered planning approach resulted in 85 individuals (70 in-person, 15 via videoconference) participating in the day-long symposium (Table 1). Most participants reported having an affiliation with an academic institution (86%, n=38 of 44 survey respondents), including across 6 CTSA sites. Thirty-five (80%) reported having only an academic affiliation while the remaining respondents (n=9/44, 20%) reported at least one nonacademic affiliation such as non-profits, start-ups, public agencies, and community organizations. All reported attaining at least a bachelor’s level educational level. Participants primarily from academic institutions were mostly master’s level (n=5/35, 14%), PhD (n=16/35, 46%) and MD (n=6/35, 17%, including those with combined degrees). There were 2 individuals with JD’s and one individual not reporting their educational level. Those participants indicating nonacademic affiliations had similar educational levels (n=1/9 with master’s level, n=8/9 with MD and/or PhD). In total, 77% (n=34/44) indicated the use of techniques or principles across 3 or more approaches. Many reported not having previously used one or more key approaches presented in the symposium. For example, 45% (n=20/44, not in table) did not indicate that they used techniques from predictive analytics. Conversely, of those that had worked in predictive analytics, 58% (n=14/24, not in tables) had not worked in community/patient engagement or social sciences. Participants’ work spanned the translational spectrum from T0 to T4 (Table 2). Participants reported work across multiple translational domains: 75% (n=33/44) reported work in 2 or more translational domains and 30% (n=13/44) in 3 or more domains.

Table 1 Symposium participants’ baseline characteristics (n=44)

Table 2 Distribution of participation across translational research domains (n=44)

Key challenges that participants identified related to conducting their work included complexity of data or techniques (61%, n=27/44), perceived relevance to clinical applications (45%, n=20/44) and data ownership/sharing (39%, n=17/44; Table 3). Barriers to collaboration included lack of funding for collaborations (61%, n=27/44), excessive time required (36%, n=16/44) and difficulty identifying relevant collaborators (34%, n=15/44). Eighty-eight percent (n=35/40, not in tables) of respondents agreed or strongly agreed that they found the symposium helpful and planned to integrate lessons learned in future work. The group was more likely after the symposium to agree that it is important to involve patients and community organizations before identifying research questions (pre-symposium: 3.67, post-symposium: 4.15; p<0.05; 1=strongly disagree to 5=strongly agree, not in tables). There was no significant change in agreement related to planning to integrate community-partnered techniques at early stages in future work (pre: 3.77, post: 3.95; p=0.15, not in tables).

Table 3 Challenges and opportunities identified by stakeholders (n=44)

Two pilot grants were awarded as part of the symposium: (1) predicting depression outcomes from electronic health record data and (2) social network analysis to predict autism outcomes. At 3-year follow-up, awardees reported a variety of products resulting from the competition including 8 new pairwise collaborations across 2 CTSA sites, 3 journal publications, and 8 symposium presentations.

Discussion

This report describes a CTSA-funded pilot adapting CPPR methods to foster team science approaches. This included partnered engagement in planning a scientific symposium and pilot funding competition. In addition, we introduced partnered methods during the symposium to develop awareness of the importance of including stakeholders in research leadership. The partnered engagement approach to planning resulted in the broad inclusion of scientists across multiple institutions and diverse translational domains, both with and without experience in predictive analytics. This represents an exposure of participants to novel fields, an important step in facilitating the formation of new team science collaborations. Amongst survey respondents, we observed a significant increase in agreement pre-symposium Versus post-symposium of the importance of involving patients and community organizations early in the research process. However, no significant shift was observed in agreement with planning to integrate community-partnered techniques. Consistent with prior reports from CTSA sites, the participants identified lack of funding as a barrier to collaboration [7]. The symposium pilot funding competition to build participatory team science in predictive analytics represented an opportunity to directly provide this financial support and resulted in multiple journal papers, conference presentations and new collaborations reported by awardees at 3-year follow-up.

Although this study has limited follow-up and small sample sizes, our findings are consistent with a recent report from the Medical University of South Carolina CTSA summarizing 5 years of scientific retreats showing that the scientific symposium format paired with funding opportunities may enhance team science collaborations as measured by co-authored, peer-reviewed publications [15]. However, as team science involves individuals across disciplines and settings, alignment of goals and effective communication across these domains (each with specific knowledge, vocabularies, and scientific approaches) continues to be cited as a challenge [16]. Our results suggest that participatory research approaches may facilitate addressing these challenges through its focus on early and broad stakeholder engagement, equal co-leadership, and support for developing a shared community.

Our approach focused on participatory engagement of primarily academic individuals across disciplines, translational domains and sites. Therefore, there was minimal engagement of patient, family, and community representatives. Inclusion of these broader stakeholders in planning and collaboration efforts would be beneficial and we recommend future efforts include larger symposiums with more involvement of these nonacademic stakeholders. In addition, this pilot was limited to 1-day event. Other groups (including CTSA sites) interested in implementing these partnered methods may consider forming a standing participatory stakeholder advisory board and ongoing activities to support collaborations and build capacity for translational team science over time, paired with longitudinal follow-up to evaluate effectiveness. Comparative studies of this partnered engagement strategy with other approaches to stimulate team science in predictive analytics may also be informative.

Future efforts in predictive health analytics may also explore two challenges identified by participants: data ownership/sharing and enhancing clinical relevance. Updating privacy and ethics guidelines may be needed to address advances in technologies, analytic techniques, and merging of large data sets that may render current methods to protect patient privacy (such data de-identification) obsolete [4, 5, 17, 18]. The relevance of these techniques for predicting individual or group-level outcomes may also be limited by under-sampling of some groups, especially low-income, under-resourced communities, due to well-documented distrust and diminished access to research opportunities. A recent World Economic Forum report describes several related recommendations including the need for transparency, trust, and engagement of broad stakeholders to mitigate risk and to ensure equitable distribution of value resulting from these efforts [19]. Trust and transparency were also cited as important factors to ensure ethical use of technologies in mental health as well [20].

Interdisciplinary collaboration and stakeholder engagement at all research stages is critical to address the unique challenges and opportunities in predictive analytics and related efforts for diverse populations. Inclusion of participatory research approaches in planning and educational activities may be an effective model for engaging broad academic disciplines and stakeholders in advancing predictive analytics in research and healthcare.

Supplementary materials

To view supplementary material for this article, please visit https://doi.org/10.1017/cts.2018.313

Acknowledgments

Symposium and collaborative efforts supported by the National Center for Advancing Translational Sciences UCLA CTSI Grant UL1TR000124. Pilot competitions supported by the Autism Intervention Research Network for Behavioral Health, HRSA Award no. UA3MC11055. A.C.A., B.C., and K.B.W. were supported through PCORI 1501-26518 and the California State Center of Excellence in Behavioral Health (SB-852). A.C.A. was supported by NIMH R01 MH093676-02S1. J.P. was supported by the National Institute on Aging and the National Center for Advancing Translational Science of the National Institutes of Health under awards K23AG049181 and UCLA CTSI KL2TR000122. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Disclosures

A.C.A. is the founder of Open Science Initiative, Arevian Technologies Inc., and Insight Health Systems Inc. S.N. is Chief Scientist at Behavioral Signal Technologies and Behavioral Informatix. All other author have no conflicts of interest to declare.

References

1. Hahn, T, Nierenberg, AA, Whitfield-Gabrieli, S. Predictive analytics in mental health: applications, guidelines, challenges and perspectives. Molecular Psychiatry 2017; 22: 3743.
2. Parikh, RB, Kakad, M, Bates, DW. Integrating predictive analytics into high-value care: the dawn of precision delivery. JAMA 2016; 315: 651652.
3. Narayanan, S, Georgiou, P. Behavioral signal processing: deriving human behavioral informatics from speech and language. Proceedings of the IEEE Institute of Electrical and Electronics Engineering 2013; 101: 12031233.
4. Lane, J, et al. Privacy, Big Data, and the Public Good. New York: Cambridge University Press, 2014.
5. Collmann, J, Matei, SA. Ethical Reasoning in Big Data: An Exploratory Analysis. Basel, Switzerland: Springer International Publishing, 2016.
6. Jones, L, Wells, K. Strategies for academic and clinician engagement in community-participatory partnered research. JAMA 2007; 297: 407410.
7. National Research Council. Enhancing the Effectiveness of Team Science. Washington, DC: The National Academies Press, 2015.
8. Bennett, L, Gadlin, H, Levine-Finley, S. Collaboration and Team Science: A Field Guide. Washington, DC: National Institutes of Health, 2010.
9. Franco, ZE, et al. A social network analysis of 140 community-academic partnerships for health: examining the healthier Wisconsin partnership program. Clinical and Translational Science 2015; 8: 311319.
10. Wooten, KC, et al. Evolution of multidisciplinary translational teams (MTTs): insights for accelerating translational innovations. Clinical and Translational Science 2015; 8: 542552.
11. Wallerstein, NB, Duran, B. Using community-based participatory research to address health disparities. Health Promotion Practice 2006; 7: 312323.
12. Alegría, M, et al. Community-based partnered research: new directions in mental health services research. Ethnicity & Disease 2011; 21(Suppl 1): 816.
13. Unertl, KM, et al. Integrating community-based participatory research and informatics approaches to improve the engagement and health of underserved populations. Journal of the American Medical Informatics Association 2016; 23: 6073.
14. Khoury, MJ, et al. The continuum of translation research in genomic medicine: how can we accelerate the appropriate integration of human genome discoveries into health care and disease prevention? Genetics in Medicine 2007; 9: 665674.
15. Ranwala D, Alberg AJ, Brady KT, Obeid JS, Davis R, & Halushka PV. Scientific retreats with “speed dating”: networking to stimulate new interdisciplinary translational research collaborations and team science. Journal of Investigative Medicine 2017; 65: 382–390.
16. Leahey, E. From solo investigator to team scientist: trends in the practice and study of research collaboration. Annual Review of Sociology 2016; 42: 81100.
17. de Montjoye, YA, et al. Identity and privacy. Unique in the shopping mall: on the reidentifiability of credit card metadata. Science (New York, NY) 2015; 347: 536539.
18. Schwartz, PM, Solove, DJ. The PII Problem: Privacy and a New Concept of Personally Identifiable Information. New York University Law Review 2011; 86: 1814.
19. Pentland, A. Personal Data: The Emergence of a New Asset Class. Geneva, Switzerland: World Economic Forum, 2011.
20. Torous, J, Roberts, LW. Needed innovation in digital health and smartphone applications for mental health. JAMA Psychiatry 2017; 74: 437438.