Skip to main content Accessibility help
×
Home

Measuring quality and outcomes of research collaborations: An integrative review

  • Beth B. Tigges (a1), Doriane Miller (a2), Katherine M. Dudding (a3), Joyce E. Balls-Berry (a4), Elaine A. Borawski (a5), Gaurav Dave (a6), Nathaniel S. Hafer (a7), Kim S. Kimminau (a8), Rhonda G. Kost (a9), Kimberly Littlefield (a10), Jackilen Shannon (a11), Usha Menon (a12) and The Measures of Collaboration Workgroup of the Collaboration and Engagement Domain Task Force, National Center for Advancing Translational Sciences, National Institutes of Health (a1) (a2) (a3) (a4) (a5) (a6) (a7) (a8) (a9) (a10) (a11) (a12)...

Abstract

Introduction:

Although the science of team science is no longer a new field, the measurement of team science and its standardization remain in relatively early stages of development. To describe the current state of team science assessment, we conducted an integrative review of measures of research collaboration quality and outcomes.

Methods:

Collaboration measures were identified using both a literature review based on specific keywords and an environmental scan. Raters abstracted details about the measures using a standard tool. Measures related to collaborations with clinical care, education, and program delivery were excluded from this review.

Results:

We identified 44 measures of research collaboration quality, which included 35 measures with reliability and some form of statistical validity reported. Most scales focused on group dynamics. We identified 89 measures of research collaboration outcomes; 16 had reliability and 15 had a validity statistic. Outcome measures often only included simple counts of products; publications rarely defined how counts were delimited, obtained, or assessed for reliability. Most measures were tested in only one venue.

Conclusions:

Although models of collaboration have been developed, in general, strong, reliable, and valid measurements of such collaborations have not been conducted or accepted into practice. This limitation makes it difficult to compare the characteristics and impacts of research teams across studies or to identify the most important areas for intervention. To advance the science of team science, we provide recommendations regarding the development and psychometric testing of measures of collaboration quality and outcomes that can be replicated and broadly applied across studies.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Measuring quality and outcomes of research collaborations: An integrative review
      Available formats
      ×

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Measuring quality and outcomes of research collaborations: An integrative review
      Available formats
      ×

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Measuring quality and outcomes of research collaborations: An integrative review
      Available formats
      ×

Copyright

This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.

Corresponding author

Address for correspondence: B. B. Tigges, MSC07 4380, 1 University of New Mexico, Albuquerque, NM 87131-0001, USA. Email: btigges@salud.unm.edu

References

Hide All
1. Sung, NS, Crowley, WF, Genel, M, et al. Central challenges facing the national clinical research enterprise. Journal of the American Medical Association 2003; 289(10): 12781287.
2. Westfall, JM, Mold, J, Fagnan, L. Practice-based research – “blue highways” on the NIH roadmap. Journal of the American Medical Association 2007; 297(4): 403406.
3. Woolf, SH. The meaning of translational research and why it matters. Journal of the American Medical Association 2008; 299(2): 211213.
4. National Center for Advancing Translational Sciences (NCATS). Strategic Goal 2: advance translational team science by fostering innovative partnerships and collaborations with a strategic array of stakeholders [Internet], 2017. https://ncats.nih.gov/strategicplan/goal2. Accessed April 25, 2019.
5. Disis, ML, Slattery, JT. The road we must take: multidisciplinary team science. Science Translational Medicine 2010; 2: 22cm9.
6. Fortunato, S, Bergstrom, CT, Börner, K, et al. Science of science. Science 2018; 359: eaao0185.
7. Jones, BF, Wuchty, S, Uzzi, B. Multi-university research teams: shifting impact, geography, and stratification in science. Science 2008; 322: 12591262.
8. Wuchty, S, Jones, BF, Uzzi, B. The increasing dominance of teams in production of knowledge. Science 2007; 316: 10361039.
9. Uzzi, B, Mukherjee, S, Stringer, M, et al. Atypical combinations and scientific impact. Science 2013; 342: 468472.
10. Larivière, V, Haustein, S, Börner, K. Long-distance interdisciplinarity leads to higher scientific impact. PLoS ONE 2015; 10(3): e0122565.
11. Basner, JE, Theisz, KI, Jensen, US, et al. Measuring the evolution and output of cross-disciplinary collaborations within the NCI Physical Sciences-Oncology Centers Network. Research Evaluation 2013; 22: 285297.
12. Hall, KL, Stokols, D, Moser, RP, et al. The collaboration readiness of transdisciplinary research teams and centers. Findings from the National Cancer Institute’s TREC year-one evaluation study. American Journal of Preventive Medicine 2008; 35(2S) :S161S172.
13. Cummings, JN, Kiesler, S, Zadeh, RB, et al. Group heterogeneity increases the risks of large group size: a longitudinal study of productivity in research groups. Psychological Science 2013; 24(6): 880890.
14. Hall, KL, Stokols, D, Stipelman, BA, et al. Assessing the value of team science. A study comparing center- and investigator-initiated grants. American Journal of Preventive Medicine 2012; 42(2): 157163.
15. Hall, KL, Vogel, AL, Stipelman, BA, et al. A four-phase model of transdisciplinary team-based research: goals, team processes, and strategies. Translational Behavioral Medicine: Practice, Policy, Research 2012; 2: 415430.
16. Salazar, MR, Lant, TK, Fiore, SM, et al. Facilitating innovation in diverse science teams through integrative capacity. Small Group Research 2012; 43(5): 527558.
17. Trochim, WM, Marcus, SE, Mâsse, LC, et al. The evaluation of large research initiatives: a participatory integrative mixed-methods approach. American Journal of Evaluation 2008; 29(1): 828.
18. Luukkonen, T, Tijssen, RJW, Persson, O, et al. The measurement of international scientific collaboration. Scientometrics 1993; 28(1): 1536.
19. Mâsse, LC, Moser, RP, Stokols, D, et al. Measuring collaboration and transdisciplinary integration in team science. American Journal of Preventive Medicine 2008; 35(2S): S151S160.
20. Milojevic, S. Principles of scientific research team formation and evolution. Proceedings of the National Academy of Sciences of the United States of America 2014; 111(11): 39843989.
21. Cooke, NJ and Hilton, ML, eds.; Committee on the Science of Team Science; Board on Behavioral, Cognitive, and Sensory Sciences; Division of Behavioral and Social Sciences and Education; National Research Council. Enhancing the effectiveness of team science. Washington, DC: The National Academies Press; 2015.
22. Nichols, LG. A topic model approach to measuring interdisciplinarity at the National Science Foundation. Scientometrics 2014; 100: 741754.
23. Oetzel, JG, Zhou, C, Duran, B, et al. Establishing the psychometric properties of constructs in a community-based participatory research conceptual model. American Journal of Health Promotion 2015; 29(5): e188e202.
24. Salas, E, Grossman, R, Hughes, AM, et al. Measuring team cohesion: observations from the science. Human Factors 2015; 57(3): 365374.
25. Stokols, D, Harvey, R, Gress, J, et al. In vivo studies of transdisciplinary scientific collaboration. Lessons learned and implications for active living research. American Journal of Preventive Medicine 2005; 28(2S2): 202213.
26. Wageman, R, Hackman, JR, Lehman, E. Team diagnostic survey: development of an instrument. The Journal of Applied Behavioral Science 2005; 41(4): 373398.
27. Wooten, KC, Rose, RM, Ostir, GV, et al. Assessing and evaluating multidisciplinary translational teams: a mixed methods approach. Evaluation and the Health Professions 2014; 37(1): 3349.
28. Misra, S, Stokols, D, Cheng, L. The transdisciplinary orientation scale: factor structure and relation to the integrative quality and scope of scientific publications. Journal of Translational Medicine and Epidemiology 2015; 3(2): 1042.
29. Souza, MT, Silva, MD, Carvalho, RD. Integrative review: what is it? How to do it? Einstein 2010; 8(1): 102106.
30. Whitehead, D, LoBiondo-Wood, G, Haber, J. Nursing and Midwifery Research: Methods and Appraisal for Evidence Based Practice. 5th ed. Chatswood NSW, Australia: Elsevier, 2016.
31. Bian, J, Xie, M, Topaloglu, U, Hudson, T, et al. Social network analysis of biomedical research collaboration networks in a CTSA institution. Journal of Biomedical Informatics 2014; 52: 130140.
32. Franco, ZE, Ahmed, SM, Maurana, CA, et al. A social network analysis of 140 community-academic partnerships for health: examining the healthier Wisconsin partnership program. Clinical and Translational Science 2015; 8(4): 311319.
33. Hughes, ME, Peeler, J, Hogenesch, JB. Network dynamics to evaluate performance of an academic institution. Science Translational Medicine 2010; 2(53): 53ps49.
34. Huang, C. Knowledge sharing and group cohesiveness on performance: an empirical study of technology R&D teams in Taiwan. Technovation 2009; 29: 786797.
35. Philbin, S. Measuring the performance of research collaborations. Measuring Business Excellence 2008; 12(3): 1623.
36. Bietz, MJ, Abrams, S, Cooper, DM, et al. Improving the odds through the Collaboration Success Wizard. Translational Behavioral Medicine: Practice, Policy, Research 2012; 2: 480486.
37. Greene, SM, Hart, G, Wagner, EH. Measuring and improving performance in multicenter research consortia. Journal of the National Cancer Institute Monographs 2005; 35: 2632.
38. Lee, S, Bozeman, B. The impact of research collaboration on scientific productivity. Social Studies of Science 2005; 35(5): 673702.
39. Mallinson, T, Lotrecchiano, GR, Schwartz, LS, et al. Pilot analysis of the Motivation Assessment for Team Readiness, Integration, and Collaboration (MATRiCx) using Rasch analysis. Journal of Investigative Medicine 2016;0:18.
40. Mazumdar, M, Messinger, S, Finkelstein, DM, et al. Evaluating academic scientists collaborating in team-based research: a proposed framework. Academic Medicine 2015; 90(10): 13021308.
41. Okamoto, J, The Centers for Population Health and Health Disparities Evaluation Working Group. Scientific collaboration and team science: a social network analysis of the centers for population health and health disparities. Translational Behavioral Medicine: Practice, Policy, Research 2015; 5: 1223.
42. National Cancer Institute, National Institutes of Health. Team Science Toolkit: TREC baseline survey [Internet], No date. https://www.teamsciencetoolkit.cancer.gov/Public/TSResourceMeasure.aspx?tid=2%26rid=36. Accessed April 25, 2019.
43. National Cancer Institute, National Institutes of Health. Team Science Toolkit: Written products protocol – TREC I study [Internet], 2006. https://www.teamsciencetoolkit.cancer.gov/Public/TSResourceMeasure.aspxtid=?2%26rid=646. Accessed April 25, 2019.
44. University of Cincinnati. Appendix D1: Researcher Survey Form [Internet], 2002. https://cctst.uc.edu/sites/default/files/cis/survey-TTURC_research.pdf. Accessed April 25, 2019.
45. Ameredes, BT, Hellmich, MR, Cestone, CM, et al. The Multidisciplinary Translational Team (MTT) Model for training and development of translational research investigators. Clinical and Translational Science 2015; 8(5): 533541.
46. Lee, YS. The sustainability of university-industry research collaboration: an empirical assessment. Journal of Technology Transfer 2000; 25: 111133.
47. Lööf, H, Broström, A. Does knowledge diffusion between university and industry increase innovativeness? Journal of Technology Transfer 2008; 33: 7390.
48. Luke, DA, Carothers, BJ, Dhand, A, et al. Breaking down silos: mapping growth of cross-disciplinary collaboration in a translational science initiative. Clinical and Translational Science 2015; 8(2): 143149.
49. Petersen, AM. Quantifying the impact of weak, strong, and super ties in scientific careers. Proceedings of the National Academy of Sciences of the United States of America 2015; 112(34): E4671E4680.
50. Stvilia, B, Hinnant, C, Schindler, K, et al. Team diversity and publication patterns in a scientific laboratory. Journal of American Society for Information Science and Technology 2011; 62(2): 270283.
51. Wang, J, Hicks, D. Scientific teams: self-assembly, fluidness, and interdependence. Journal of Informetrics 2014; 9(1): 197207.
52. Lee, Y, Walsh, JP, Wang, J. Creativity in scientific teams: unpacking novelty and impact. Research Policy 2014; 44: 684697.
53. Hager, K, St Hill, C, et al. Development of an interprofessional and interdisciplinary collaborative research practice for clinical faculty. Journal of Interprofessional Care 2016; 30(2): 265267.
54. Bieschke, K, Bishop, R, Garcia, V. The utility of the research self-efficacy scale. Journal of Career Assessment 1996; 4: 5975.
55. Hanel, P, St-Pierre, M. Industry-university collaboration by Canadian manufacturing firms. Journal of Technology Transfer 2006; 31: 485499.
56. Armstrong, A, Jackson-Smith, D. Forms and levels of integration: evaluation of an interdisciplinary team-building project. Journal of Research Practice 2013; 9(1): Article M1.
57. Vogel, AL, Stipelman, BA, Hall, KL, et al. Pioneering the transdisciplinary team science approach: lessons learned from the National Cancer Institute grantees. Journal of Translational Medicine and Epidemiology 2014; 2(2): 1027.
58. Aguilar-Gaxiola, S, Ahmed, S, Franco, Z, et al. Toward a unified taxonomy of health indicators: academic health centers and communities working together to improve population health. Academic Medicine 2014; 89(4): 564572.
59. Kane, M, Trochim, WM. Evaluation of large initiatives of scientific research at the National Institutes of Health. In: Presentation at the American Evaluation Association Conference. Portland, Oregon, USA. November 4, 2006.
60. Granner, ML, Sharpe, PA. Evaluating community coalition characteristics and functioning: a summary of measurement tools. Health Education Research 2004; 19(5): 514532.
61. Sandoval, JA, Lucero, J, Oetzel, J, et al. Process and outcome constructs for evaluating community-based participatory research projects: a matrix of existing measures. Health Education Research 2012; 27(4): 680690.

Keywords

Measuring quality and outcomes of research collaborations: An integrative review

  • Beth B. Tigges (a1), Doriane Miller (a2), Katherine M. Dudding (a3), Joyce E. Balls-Berry (a4), Elaine A. Borawski (a5), Gaurav Dave (a6), Nathaniel S. Hafer (a7), Kim S. Kimminau (a8), Rhonda G. Kost (a9), Kimberly Littlefield (a10), Jackilen Shannon (a11), Usha Menon (a12) and The Measures of Collaboration Workgroup of the Collaboration and Engagement Domain Task Force, National Center for Advancing Translational Sciences, National Institutes of Health (a1) (a2) (a3) (a4) (a5) (a6) (a7) (a8) (a9) (a10) (a11) (a12)...

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed