Skip to main content Accessibility help
×
Home

Information:

  • Access
  • Open access

Actions:

      • Send article to Kindle

        To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Measuring quality and outcomes of research collaborations: An integrative review
        Available formats
        ×

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Measuring quality and outcomes of research collaborations: An integrative review
        Available formats
        ×

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Measuring quality and outcomes of research collaborations: An integrative review
        Available formats
        ×
Export citation

Abstract

Introduction:

Although the science of team science is no longer a new field, the measurement of team science and its standardization remain in relatively early stages of development. To describe the current state of team science assessment, we conducted an integrative review of measures of research collaboration quality and outcomes.

Methods:

Collaboration measures were identified using both a literature review based on specific keywords and an environmental scan. Raters abstracted details about the measures using a standard tool. Measures related to collaborations with clinical care, education, and program delivery were excluded from this review.

Results:

We identified 44 measures of research collaboration quality, which included 35 measures with reliability and some form of statistical validity reported. Most scales focused on group dynamics. We identified 89 measures of research collaboration outcomes; 16 had reliability and 15 had a validity statistic. Outcome measures often only included simple counts of products; publications rarely defined how counts were delimited, obtained, or assessed for reliability. Most measures were tested in only one venue.

Conclusions:

Although models of collaboration have been developed, in general, strong, reliable, and valid measurements of such collaborations have not been conducted or accepted into practice. This limitation makes it difficult to compare the characteristics and impacts of research teams across studies or to identify the most important areas for intervention. To advance the science of team science, we provide recommendations regarding the development and psychometric testing of measures of collaboration quality and outcomes that can be replicated and broadly applied across studies.

Introduction

Translating basic science discoveries into demonstrated improvements in public health requires a research team from diverse backgrounds [13]. The US National Institutes of Health National Center for Advancing Translational Sciences recognized this need by establishing a strategic goal to advance translational team science by fostering innovative partnerships and diverse collaborations [4]. In the health sciences, there is significant interest in translational research and moving more quickly from single-study efficacy trials to effective, generalizable interventions in health care practice. Foundational to this body of literature is the assumption that cross-disciplinary research teams speed the process of translational research [5].

Analyses of trends in scientific publications suggest that major advances in biological, physical, and social science are produced by research teams; that the work of these teams is cited more often than the work of individual researchers; and that, in the long term, the work has greater scientific impact [69]. In addition, cross-disciplinary diversity is assumed to lead to greater innovation [10]. These observations have become the cornerstone of the translational science movement in the health sciences.

Implementing team science can be challenging. Multiple authors have noted that working in collaboration can be more expensive and labor intensive than working alone [11,12]. Noted trade-offs include added time and effort to communicate with diverse collaborators, conflicts arising from different goals and assumptions, and increased start-up time with its resulting delay in productivity [1317]. These opportunity costs may be acceptable if the outcomes of research collaborations can accelerate knowledge or answer the complex health questions faced by today’s society.

To test the assumption that research collaboration leads to greater productivity, we need to accurately measure the characteristics of research teams and their outcomes and be able to compare results across teams [6,12,15,1827]. Although different measures have so far shown that collaborations are beneficial, operational definitions of variables that may influence conclusions (construct validity) are varied, complicating interpretation of results. Despite some exceptions [12,19,23,28], there is a lack of attention to the development and psychometric testing of reliable and valid measures of collaboration. As an initial step, it would be useful to have an overview of the current state of the science in the measurement of research collaborations. In this article, we report the results of an integrative review of the literature, looking for reliable and valid measures that describe the quality and outcomes of research collaborations.

Materials and Methods

We conducted two reviews. The first focused on measures of collaboration quality, defined as measures of interactions or processes of the team during the collaboration. The second review focused on outcomes of the collaboration (e.g., publications, citations). We used an integrative review approach. An integrative review is a specific type of review that applies a comprehensive methodology involving a combination of different approaches to summarize past research related to a particular topic, including both experimental and non-experimental studies, and reach conclusions [29,30].

Our research team brainstormed keyword combinations and, based on expert opinion, agreed on final sets of keywords that were comprehensive enough to cover the topics fully but not so broad as to include non-relevant literature. For the review of collaboration quality, these keywords were “measure/measurement” combined with the following terms: community engagement, community engaged research, collaboration, community academic partnership, team science, regulatory collaboration, industry collaboration, public–private partnership (focus on research). For the review associated with collaboration outcomes, the word “outcomes” was added to the above search terms. Our intention was to include all types of research collaborations, including partnerships between academic and other community, governmental, and industry partners. The following keywords were considered, tested in preliminary searches, and eliminated by group consensus as being too broad for our purpose: consortium collaboration, public health and medicine collaboration, patient advocacy group collaboration, and coalition. Measures of collaboration related to clinical care, education, and program delivery collaborations were excluded from this review.

Quality and outcome measures were identified using both a literature review and an environmental scan. We conducted searches using the standard databases PubMed, the Comprehensive Index to Nursing and Allied Health Literature, and PsychInfo, as well as searched EMBASE, Google Scholar, Scopus, and websites recommended by members of the research team. After duplicates and articles that were not focused on a specific scale or measure of research collaboration were eliminated, team members reviewed a final list of 25 publications for the measures of collaboration quality, including 4 articles describing social network analyses, and 42 publications for measures of collaboration outcome. All publications were published prior to 2017. Figs. 1 and 2 provide flow diagrams of how articles were selected to be included in both reviews.

Fig. 1. Flow diagram of publications included in the final collaboration quality review.

Fig. 2. Flow diagram of publications included in the final collaboration outcomes review.

At least two members of the research team reviewed each article using a standard data abstraction form that included the name of the measure/outcome; construct being measured; sample; and details about the measure, including operational definition, number of items, response options, reliability, validity, and other evidence for supporting its use. Reviewers were also asked to make a judgment as to whether the article included a measure of the collaboration quality (or outcomes or products) of the scientific/research collaborations; both reviews had a rater agreement of 99%. Differences in reviews were resolved through consensus after discussions with a third reviewer.

Results

Quality Measures

We identified 44 measures of research collaboration quality from the 15 publications included in the final summary analyses (see Fig. 1). The specifics of each measure are detailed in Table 1. Three articles were not included in Table 1 because they all used social network analysis [3133]. Four articles covered 80% of the measures identified [12,19,23,34].

Table 1. Measures of research collaboration quality

NR, not reported; CRN, Cancer Research Network; NCI, US National Cancer Institute; TREC, Transdisciplinary Research on Energetics and Cancer; ICC, intraclass correlation coefficients; NSF, US National Science Foundation; DOE, US Department of Energy; MATRICx, Motivation Assessment for Team Readiness, Integration and Collaboration; NA, not applicable; TTURC, Transdisciplinary Tobacco Use Research Center; PI, principal investigator; PD, project director; CBPR, community-based participatory research.

a Details obtained by cross-referencing article (TREC Baseline survey) from https://www.teamsciencetoolkit.cancer.gov/Public/TSResourceMeasure.aspx?tid=2%26rid=36 [42].

b Detail obtained by cross-referencing article (NCI TREC Written Products Protocol 2006-09-27) from https://www.teamsciencetoolkit.cancer.gov/Public/TSResourceMeasure.aspx?tid=2%26rid=646 [43].

c Details obtained by cross-referencing article (TTURC Researcher Survey 2002) from https://cctst.uc.edu/sites/default/files/cis/survey-TTURC_research.pdf [44].

d Original instrument shown at http://cpr.unm.edu/research-projects/cbpr-project/index.html--scroll to 2. Quantitative Measures – “Key Informant” and “Community Engagement” survey instruments. Developmental work on measures from Oetzel et al. (2015) continues in an NIH NINR R01 (Wallerstein [PI] 2015-2020 “Engage for Equity” Study; see http://cpr.unm.edu/research-projects/cbpr-project/cbpr-e2.html).

The number of items per measure ranged from 1 to 48, with 77% having less than 10 items per measure. A few articles reported on measures that covered several domains. As shown in Table 1, we have included each domain measure separately if it was reported as an independent scale with its own individual psychometric properties.

Reliability was reported for 35 measures, not reported for four measures, and not applicable for five measures (single-item, self-reported frequency counts, or qualitative responses). Reliability measures were most frequently Cronbach’s alphas for internal consistency reliability, but also included intraclass correlation coefficients, inter-rater correlations, and, when Rasch analysis was used, person separation reliability. Test–retest reliability was never reported. Cronbach’s alpha statistics were >0.70 for 86% of the measures using that metric. Some form of validity was reported on 40 measures and typically included exploratory (n = 8) and/or confirmatory factor analysis (n = 26). Convergent or discriminant validity was evident for 38 measures but was based on study results, as interpreted by our reviewers, rather than identified by the authors as a labeled multitrait–multimethod matrix analysis of construct validity. Twelve measures had convergent or discriminant validity only, without any further exploration of validity. Face validity and content validity were reported for five measures, along with other analyses of validity.

Outcome Measures

We identified 89 outcome measures from the 24 publications included in the final summary analyses (see Fig. 2). Characteristics of each measure are detailed in Table 2. Three publications included over 44 (49%) of the measures identified [17,23,35]. However, only two of those [17,23] included measures tested in actual studies; the remaining article [35] included only recommendations for specific measures.

Table 2. Measures of research collaboration outcomes

NA, not applicable; NR, not reported; NIH, National Institutes of Health; CTSA, Clinical and Translational Science Award; NSF, National Science Foundation; ISI, Institute for Scientific Information; DOE, US Department of Energy; SCI, Science Citation Index; CV, curriculum vitae; NCI, National Cancer Institute; IF, impact factor; TTURC, Transdisciplinary Tobacco Use Research Centers; NHMFL, National High Magnetic Field Laboratory;RTI, relative team impact; TREC, Transdisciplinary Research on Energetics and Cancer; PI, principal investigator; PD, project director; CBPR, community-based participatory research.

a Details obtained by cross-referencing article (TTURC Researcher Survey 2002) from https://cctst.uc.edu/sites/default/files/cis/survey-TTURC_research.pdf [44].

b Details obtained by cross-referencing article (TREC Baseline survey) from https://www.teamsciencetoolkit.cancer.gov/Public/TSResourceMeasure.aspx?tid=2%26rid=36 [42].

c Original instrument available at http://cpr.unm.edu/research-projects/cbpr-project/index.html--scroll to 2 (Quantitative Measures – “Key Informant” and “Community Engagement” survey instruments). Developmental work on measures from Oetzel et al. (2015) continues in an NIH NINR R01 (Wallerstein [PI], 2015–2020 “Engage for Equity” study (see http://cpr.unm.edu/research-projects/cbpr-project/cbpr-e2.html).

d Details obtained by cross-referencing article (TTURC Researcher Survey 2002) from https://cctst.uc.edu/sites/default/files/cis/survey-TTURC_research.pdf [44] and from Kane and Trochim [59].

e Details obtained by cross-referencing article (NCI TREC Written Products Protocol 2006-09-27) from https://www.teamsciencetoolkit.cancer.gov/Public/TSResourceMeasure.aspx?tid=2%26rid=646 [43].

Measures were broadly classified into one of the six different categories, reflected in Table 2: (1) counts or numerical representations of products (e.g., number of publications; 38 measures); (2) quality indicators of counted products (e.g., journal impact factor; 7 measures); (3) self-reported perceptions of outcomes (e.g., perceived productivity; 32 measures); (4) peer-reviewed perceptions of outcomes (e.g., progress on the development of interventions; 5 measures); (5) qualitative descriptions of outcomes (e.g., descriptive data collected by interview; 6 measures); and (6) health indicators/outcomes (e.g., life expectancy; 1 overall measure with 60 different indicators). The number of items per measure ranged from a single count to a 99-item scale, with over 50% of the measures composed of a single count, number, or rating of a single item.

Twenty-three of the 89 measures were recommendations on measures and had no reported reliability or validity as would be expected [35]. For the remaining 66 measures, only 16 reported assessments of reliability. Nine of 24 measures in the self-reported perceptions category included Cronbach’s alpha >0.70, showing internal consistency reliability. Six measures (3 of 24 in the counts of products category and 3 of 4 in the peer-reviewed category) had inter-rater agreement described; all were over 80%. One measure in the peer-reviewed category reported inter-rater reliability of r = 0.24–0.69. Of these 16 measures with reported reliability, nine had some form of validity described: confirmatory factor analysis (6 measures) and convergent validity (3 measures). Of the remaining 50 measures without reliability data, five had some type of convergent validity described and one was supported by principal component analysis. Once again, convergent validity was not formally labeled as such but was evident in terms of correlations between the measure under study and other relevant variables.

Discussion

Quality Measures

Overall, there are a relatively large number of scales, some of them robust, that have been used to measure the quality or process of research collaborations (e.g., trust, frequency of collaboration). However, many scales have not been extensively used and have been subjected to relatively little repeated psychometric study and analysis. Most have been developed in support of a particular research project rather than with the intent of becoming a standard indicator or scale for the field. Although calculated across multiple organizations, estimates of reliability and/or validity were often study specific as well. Reports of effect sizes (sensitivity or responsiveness) were rare and limited to correlations, and construct validity has not been explored beyond exploratory or confirmatory factor analyses. Given this dearth of replicated psychometric data, it is not surprising that widely accepted, standard scales have not emerged to date. Wide-scale testing of measures of collaboration is essential to establish reliability, validity, and sensitivity or responsiveness across settings and samples.

Scales developed to date have been primarily focused on group dynamics (including the quality of interpersonal interactions, trust, and communication). Although these are important factors, few measurements have been made of how well a team functions (such as leadership styles) and the degree to which the team’s work is viewed as synergistic, integrative, or otherwise more valuable than would occur in a more siloed setting. Oetzel et al.’s [23] beginning psychometric work provides an example of some of these types of measures. This is in contrast to the numerous available (or under development) scales to measure attitudes toward collaborations and quality of collaborations that exist at specific institutions.

Despite these limitations, two sets of measures deserve note. First, those reported by Hall et al. [12] and Mâsse et al. [19] as measures of collaborations in National Cancer Institute-funded Transdisciplinary Tobacco Use Research Centers have been used more extensively than many of the other scales in this review as indicators of collaboration quality among academic partners (although relatively little additional psychometric data have been reported beyond initial publications). Second, the measures reported by Oetzel et al. [23] are unique in that they are scales to assess research quality involving collaborations between academics and communities, agencies, and/or community-based organizations. They are also unique in representing responses from over 200 research partnerships across the USA. This review did not distinguish between partnerships (e.g., involving just two partnering organizations) and coalitions (involving multiple organizations).

Outcome Measures

Similar to measures of collaboration quality, little agreement exists as to how to best measure outcomes of research collaborations. By far, the most common type of measurement is a simple count of products over a set period of time (e.g., publications, grants, and/or patents). Interestingly, the procedures used for counting or calculating these products are rarely reported and therefore are not replicable. In addition, published reports infrequently include any type of verification of counts, leaving the reliability of such counts or calculations in question.

The second most common type of measure is the use of self-reported scales to quantify the researchers’ perceptions of collaboration outcomes. These include measures of perceived productivity or progress, changes in relationships with partners, increased capacity, and sustainability. Few of these measures, with the exception of the psychometric works of Hall et al. [12] and Oetzel et al. [23], have documented reliability and validity. In general, despite a relatively large number of scales, most of these were not developed for the purpose of becoming standard indicators or measures and most have had little psychometric study or replication.

Efforts to measure the quality of counted products, such as consideration of citation percentiles, journal impact factors, or field performance indicators, offer important alternatives in the quantity versus quality debate and actually may be useful for evaluating the long-term scientific impact of collaborative outcomes. Likewise, peer-reviewed ratings of outcomes based on reviews of proposals or progress reports could provide more neutral and standardized measures of collaboration impact. Both of these categories of measures are used infrequently but could have significant influence if applied more widely in the evaluation of collaborative work. However, further work on a reliable rating’s scale for use in peer review is needed before it is able to provide comparable results across studies.

Recommendations

Remarkably, the results of this review, which defines research collaborations to include different types of collaborative partnerships, are very similar to reviews of measures of community coalitions [60] and community-based participatory research [61] conducted 15 and 7 years ago, respectively. Both of those studies concluded that there are few reliable and valid measures. In the intervening years, some progress has been made as noted [see Refs. 12, 19, 23 as examples]. Based on this observation and our findings in this study, we offer six recommendations to advance the field of team science: (1) We must pay careful attention and devote resources to the development and psychometric testing of measures of research collaboration quality and outcomes that can be replicated and broadly applied. Measures listed in this review with solid initial reliability and validity indicators provide reasonable starting points for continued development; however, measures of other constructs will also be necessary. (2) To establish validity for use in different populations and settings, designed measures should be tested across various research partner and stakeholder relationships (e.g., academia, industry, government, patient, community, and advocacy groups). (3) When evaluating outcomes, it is critical that we focus on both the quality and quantity of products and the use of rating scales for peer review. (4) The sensitivity and responsiveness of measures to interventions should be evaluated as an additional psychometric property. (5) Publications reporting on assessments of collaborations should include a clear description of the measures used; the reliability, validity, and sensitivity or responsiveness of the measures; and a statement on their generalizability. (6) Reports incorporating the use of narrowly applicable measures should include a justification for not using a more broadly applicable measure.

Conclusions

Although a few studies have conducted exemplary psychometric analyses of some measures of both collaboration quality and outcomes, most existing measures are not well-defined; do not have well-documented reliability, validity, or sensitivity or responsiveness (quality measures); and have not been replicated. Construct validity, in particular, requires further exploration. Most of the reported measures were developed for a single project and were not tested across projects or types of teams. Published articles do not use consistent measures and often do not provide operational definitions of the measures that were used. As a result of all of these factors, it is difficult to compare the characteristics and impact of research collaborations across studies.

Team science and the study of research collaborations are becoming better and more rigorous fields of inquiry; however, to truly understand the reasons that some teams succeed and others fail, and to develop effective interventions to facilitate team effectiveness, accurate and precise measurements of the characteristics and the outcomes of the collaborations are needed to further translational science and the concomitant improvements in public health.

Acknowledgements

This work was supported by the National Center for Advancing Translational Sciences, National Institutes of Health, Grant numbers: UL1 TR001449 (BBT), UL1 TR000430 (DM), UL1 TR002389 (DM), UL1 TR002377 (JBB), UL1 TR000439 (EAB), UL1 TR002489 (GD), UL1 TR001453 (NSH), UL1 TR002366 (KSK), UL1 TR001866 (RGK), UL1 TR003096 (KL), UL1 TR000128 (JS), and U24 TR002269 (University of Rochester Center for Leading Innovation and Collaboration Coordinating Center for the Clinical and Translational Science Awards Program).

Disclosures

The authors have no conflicts of interest to declare.

Disclaimer

The views expressed in this article are the responsibility of the authors and do not necessarily represent the position of the National Center for Advancing Translational Sciences, the National Institutes of Health, or the US Department of Health and Human Services.

References

1. Sung, NS, Crowley, WF, Genel, M, et al. Central challenges facing the national clinical research enterprise. Journal of the American Medical Association 2003; 289(10): 12781287.
2. Westfall, JM, Mold, J, Fagnan, L. Practice-based research – “blue highways” on the NIH roadmap. Journal of the American Medical Association 2007; 297(4): 403406.
3. Woolf, SH. The meaning of translational research and why it matters. Journal of the American Medical Association 2008; 299(2): 211213.
4. National Center for Advancing Translational Sciences (NCATS). Strategic Goal 2: advance translational team science by fostering innovative partnerships and collaborations with a strategic array of stakeholders [Internet], 2017. https://ncats.nih.gov/strategicplan/goal2. Accessed April 25, 2019.
5. Disis, ML, Slattery, JT. The road we must take: multidisciplinary team science. Science Translational Medicine 2010; 2: 22cm9.
6. Fortunato, S, Bergstrom, CT, Börner, K, et al. Science of science. Science 2018; 359: eaao0185.
7. Jones, BF, Wuchty, S, Uzzi, B. Multi-university research teams: shifting impact, geography, and stratification in science. Science 2008; 322: 12591262.
8. Wuchty, S, Jones, BF, Uzzi, B. The increasing dominance of teams in production of knowledge. Science 2007; 316: 10361039.
9. Uzzi, B, Mukherjee, S, Stringer, M, et al. Atypical combinations and scientific impact. Science 2013; 342: 468472.
10. Larivière, V, Haustein, S, Börner, K. Long-distance interdisciplinarity leads to higher scientific impact. PLoS ONE 2015; 10(3): e0122565.
11. Basner, JE, Theisz, KI, Jensen, US, et al. Measuring the evolution and output of cross-disciplinary collaborations within the NCI Physical Sciences-Oncology Centers Network. Research Evaluation 2013; 22: 285297.
12. Hall, KL, Stokols, D, Moser, RP, et al. The collaboration readiness of transdisciplinary research teams and centers. Findings from the National Cancer Institute’s TREC year-one evaluation study. American Journal of Preventive Medicine 2008; 35(2S) :S161S172.
13. Cummings, JN, Kiesler, S, Zadeh, RB, et al. Group heterogeneity increases the risks of large group size: a longitudinal study of productivity in research groups. Psychological Science 2013; 24(6): 880890.
14. Hall, KL, Stokols, D, Stipelman, BA, et al. Assessing the value of team science. A study comparing center- and investigator-initiated grants. American Journal of Preventive Medicine 2012; 42(2): 157163.
15. Hall, KL, Vogel, AL, Stipelman, BA, et al. A four-phase model of transdisciplinary team-based research: goals, team processes, and strategies. Translational Behavioral Medicine: Practice, Policy, Research 2012; 2: 415430.
16. Salazar, MR, Lant, TK, Fiore, SM, et al. Facilitating innovation in diverse science teams through integrative capacity. Small Group Research 2012; 43(5): 527558.
17. Trochim, WM, Marcus, SE, Mâsse, LC, et al. The evaluation of large research initiatives: a participatory integrative mixed-methods approach. American Journal of Evaluation 2008; 29(1): 828.
18. Luukkonen, T, Tijssen, RJW, Persson, O, et al. The measurement of international scientific collaboration. Scientometrics 1993; 28(1): 1536.
19. Mâsse, LC, Moser, RP, Stokols, D, et al. Measuring collaboration and transdisciplinary integration in team science. American Journal of Preventive Medicine 2008; 35(2S): S151S160.
20. Milojevic, S. Principles of scientific research team formation and evolution. Proceedings of the National Academy of Sciences of the United States of America 2014; 111(11): 39843989.
21. Cooke, NJ and Hilton, ML, eds.; Committee on the Science of Team Science; Board on Behavioral, Cognitive, and Sensory Sciences; Division of Behavioral and Social Sciences and Education; National Research Council. Enhancing the effectiveness of team science. Washington, DC: The National Academies Press; 2015.
22. Nichols, LG. A topic model approach to measuring interdisciplinarity at the National Science Foundation. Scientometrics 2014; 100: 741754.
23. Oetzel, JG, Zhou, C, Duran, B, et al. Establishing the psychometric properties of constructs in a community-based participatory research conceptual model. American Journal of Health Promotion 2015; 29(5): e188e202.
24. Salas, E, Grossman, R, Hughes, AM, et al. Measuring team cohesion: observations from the science. Human Factors 2015; 57(3): 365374.
25. Stokols, D, Harvey, R, Gress, J, et al. In vivo studies of transdisciplinary scientific collaboration. Lessons learned and implications for active living research. American Journal of Preventive Medicine 2005; 28(2S2): 202213.
26. Wageman, R, Hackman, JR, Lehman, E. Team diagnostic survey: development of an instrument. The Journal of Applied Behavioral Science 2005; 41(4): 373398.
27. Wooten, KC, Rose, RM, Ostir, GV, et al. Assessing and evaluating multidisciplinary translational teams: a mixed methods approach. Evaluation and the Health Professions 2014; 37(1): 3349.
28. Misra, S, Stokols, D, Cheng, L. The transdisciplinary orientation scale: factor structure and relation to the integrative quality and scope of scientific publications. Journal of Translational Medicine and Epidemiology 2015; 3(2): 1042.
29. Souza, MT, Silva, MD, Carvalho, RD. Integrative review: what is it? How to do it? Einstein 2010; 8(1): 102106.
30. Whitehead, D, LoBiondo-Wood, G, Haber, J. Nursing and Midwifery Research: Methods and Appraisal for Evidence Based Practice. 5th ed. Chatswood NSW, Australia: Elsevier, 2016.
31. Bian, J, Xie, M, Topaloglu, U, Hudson, T, et al. Social network analysis of biomedical research collaboration networks in a CTSA institution. Journal of Biomedical Informatics 2014; 52: 130140.
32. Franco, ZE, Ahmed, SM, Maurana, CA, et al. A social network analysis of 140 community-academic partnerships for health: examining the healthier Wisconsin partnership program. Clinical and Translational Science 2015; 8(4): 311319.
33. Hughes, ME, Peeler, J, Hogenesch, JB. Network dynamics to evaluate performance of an academic institution. Science Translational Medicine 2010; 2(53): 53ps49.
34. Huang, C. Knowledge sharing and group cohesiveness on performance: an empirical study of technology R&D teams in Taiwan. Technovation 2009; 29: 786797.
35. Philbin, S. Measuring the performance of research collaborations. Measuring Business Excellence 2008; 12(3): 1623.
36. Bietz, MJ, Abrams, S, Cooper, DM, et al. Improving the odds through the Collaboration Success Wizard. Translational Behavioral Medicine: Practice, Policy, Research 2012; 2: 480486.
37. Greene, SM, Hart, G, Wagner, EH. Measuring and improving performance in multicenter research consortia. Journal of the National Cancer Institute Monographs 2005; 35: 2632.
38. Lee, S, Bozeman, B. The impact of research collaboration on scientific productivity. Social Studies of Science 2005; 35(5): 673702.
39. Mallinson, T, Lotrecchiano, GR, Schwartz, LS, et al. Pilot analysis of the Motivation Assessment for Team Readiness, Integration, and Collaboration (MATRiCx) using Rasch analysis. Journal of Investigative Medicine 2016;0:18.
40. Mazumdar, M, Messinger, S, Finkelstein, DM, et al. Evaluating academic scientists collaborating in team-based research: a proposed framework. Academic Medicine 2015; 90(10): 13021308.
41. Okamoto, J, The Centers for Population Health and Health Disparities Evaluation Working Group. Scientific collaboration and team science: a social network analysis of the centers for population health and health disparities. Translational Behavioral Medicine: Practice, Policy, Research 2015; 5: 1223.
42. National Cancer Institute, National Institutes of Health. Team Science Toolkit: TREC baseline survey [Internet], No date. https://www.teamsciencetoolkit.cancer.gov/Public/TSResourceMeasure.aspx?tid=2%26rid=36. Accessed April 25, 2019.
43. National Cancer Institute, National Institutes of Health. Team Science Toolkit: Written products protocol – TREC I study [Internet], 2006. https://www.teamsciencetoolkit.cancer.gov/Public/TSResourceMeasure.aspxtid=?2%26rid=646. Accessed April 25, 2019.
44. University of Cincinnati. Appendix D1: Researcher Survey Form [Internet], 2002. https://cctst.uc.edu/sites/default/files/cis/survey-TTURC_research.pdf. Accessed April 25, 2019.
45. Ameredes, BT, Hellmich, MR, Cestone, CM, et al. The Multidisciplinary Translational Team (MTT) Model for training and development of translational research investigators. Clinical and Translational Science 2015; 8(5): 533541.
46. Lee, YS. The sustainability of university-industry research collaboration: an empirical assessment. Journal of Technology Transfer 2000; 25: 111133.
47. Lööf, H, Broström, A. Does knowledge diffusion between university and industry increase innovativeness? Journal of Technology Transfer 2008; 33: 7390.
48. Luke, DA, Carothers, BJ, Dhand, A, et al. Breaking down silos: mapping growth of cross-disciplinary collaboration in a translational science initiative. Clinical and Translational Science 2015; 8(2): 143149.
49. Petersen, AM. Quantifying the impact of weak, strong, and super ties in scientific careers. Proceedings of the National Academy of Sciences of the United States of America 2015; 112(34): E4671E4680.
50. Stvilia, B, Hinnant, C, Schindler, K, et al. Team diversity and publication patterns in a scientific laboratory. Journal of American Society for Information Science and Technology 2011; 62(2): 270283.
51. Wang, J, Hicks, D. Scientific teams: self-assembly, fluidness, and interdependence. Journal of Informetrics 2014; 9(1): 197207.
52. Lee, Y, Walsh, JP, Wang, J. Creativity in scientific teams: unpacking novelty and impact. Research Policy 2014; 44: 684697.
53. Hager, K, St Hill, C, et al. Development of an interprofessional and interdisciplinary collaborative research practice for clinical faculty. Journal of Interprofessional Care 2016; 30(2): 265267.
54. Bieschke, K, Bishop, R, Garcia, V. The utility of the research self-efficacy scale. Journal of Career Assessment 1996; 4: 5975.
55. Hanel, P, St-Pierre, M. Industry-university collaboration by Canadian manufacturing firms. Journal of Technology Transfer 2006; 31: 485499.
56. Armstrong, A, Jackson-Smith, D. Forms and levels of integration: evaluation of an interdisciplinary team-building project. Journal of Research Practice 2013; 9(1): Article M1.
57. Vogel, AL, Stipelman, BA, Hall, KL, et al. Pioneering the transdisciplinary team science approach: lessons learned from the National Cancer Institute grantees. Journal of Translational Medicine and Epidemiology 2014; 2(2): 1027.
58. Aguilar-Gaxiola, S, Ahmed, S, Franco, Z, et al. Toward a unified taxonomy of health indicators: academic health centers and communities working together to improve population health. Academic Medicine 2014; 89(4): 564572.
59. Kane, M, Trochim, WM. Evaluation of large initiatives of scientific research at the National Institutes of Health. In: Presentation at the American Evaluation Association Conference. Portland, Oregon, USA. November 4, 2006.
60. Granner, ML, Sharpe, PA. Evaluating community coalition characteristics and functioning: a summary of measurement tools. Health Education Research 2004; 19(5): 514532.
61. Sandoval, JA, Lucero, J, Oetzel, J, et al. Process and outcome constructs for evaluating community-based participatory research projects: a matrix of existing measures. Health Education Research 2012; 27(4): 680690.