ABSTRACT IMPACT: This work will help assess the effectiveness of the mentored career development programs. OBJECTIVES/GOALS: There is increased attention on assessing the impact of the CTSA in building a research workforce through mentored career development programs. We propose using data visualization to assess and communicate the impact of the programs on the scholars career development. METHODS/STUDY POPULATION: Evaluators from two CTSAs collaborated to visualize the KL2 data such as demographics, scholarly productivity (publications, grants, intellectual property), and time to promotion that is already tracked through REDCap at their institutions. Excel, Tableau, and Microsoft PowerBi were then used to generate trends in scholarly productivity over time. The goal was to compare how different tools can be used to visualize bibliometric data, based on what is available at the respective institutions. RESULTS/ANTICIPATED RESULTS: Longitudinal visual summary reports were produced for the entire program as well individual scholar progress. These reports can be used to identify trends such as how long after program completion do participants achieve their next milestone, what type of milestones are achieved, when in their career is their scholarly productivity the highest, etc. Answers to these questions could tell a story of the effectiveness of a mentored development program in the participants’ career. It can also highlight gaps and areas of opportunities that the program must address, either by adapting their curriculum or clarifying their intended outcomes. DISCUSSION/SIGNIFICANCE OF FINDINGS: Data visualization provides better understanding of the impact of the training programs on the scholars career development. Such insights are otherwise missing when evaluations are only focused on the percentage of scholars who were still engaged in research after completion of the program.

ABSTRACT IMPACT: This work will inform the ongoing development of adaptive capacity and preparedness of the CTSA Program and other clinical and translational research organizations in their quest of improving processes that drive outcomes and impacts, shaping effective programs and services, and strengthening their emergency readiness and sustainability. OBJECTIVES/GOALS: -Share the progress and preliminary findings of an ‘Adaptive Capacity and Preparedness of CTSA Hubs’ CTSA Working Group; -Improve our awareness and understanding of the efficient and effective changes helping CTSA hubs build robust capacity to address METHODS/STUDY POPULATION: A multi-case study including: - Triangulating multiple sources of information and mixed methods (survey/interviews of research administrators, researchers, evaluators, and other key stakeholders), literature review, document and M&E system information analysis, and expert review; - Describing CTSA hubs’ experiences as related to research implementation, translation, and support during the time of emergency; - Administering a comprehensive survey of the CTSAs addressing their challenges, lessons learned, and practices that work in various program components/areas. Data collection includes aggregate and cross-sectional data, with representation based on CTSA size, maturity, and population density. RESULTS/ANTICIPATED RESULTS: The described approach shows sound promise to investigate and share strategies and best practices for building adaptive capacity and preparedness of CTSAs -- across various scientific sectors, translational research spectrum, and the goals outlined by NCATS for the CTSA program. The anticipated results of this research will include the identified/shared innovative solutions and lessons learned for this rapidly emerging, high-priority clinical and translational science issue. ‘High-quality lessons learned’ are those that represent principles extrapolated from multiple sources and triangulated to increase transferability to new contexts and situations. DISCUSSION/SIGNIFICANCE OF FINDINGS: The project provides useful knowledge and tools to research organizations and stakeholders across multiple disciplines -- for mitigating the impact of the COVID-19 disaster via effective adjusting programs, practices, and processes, and building capacity for future successful, ‘emergency ready and responsive’ research and training.

OBJECTIVES/GOALS: The North Carolina Translational and Clinical Sciences Institute (NC TraCS) supports faculty and staff in carrying out clinical and translational research at UNC-Chapel Hill. To better understand customer satisfaction and impact, a survey was administered among NC TraCS users. METHODS/STUDY POPULATION: NC TraCS has 13 program areas that range from Biostatistics to Community and Stakeholder Engagement. These programs provide services to faculty, staff, students, and outside researchers in the area of clinical and translational science. A customer feedback survey was administered in Spring 2019 to anyone who had used at least one NC TraCS service between March 1st, 2017 and February 28th, 2019. A total of 856 survey invitations were sent. The survey included questions around users’ perception of the ease of access, helpfulness, outcome, and promptness of the services received using 6-point Likert scale. The survey also addressed career impact, communications, and suggestions for improvement. RESULTS/ANTICIPATED RESULTS: We received 268 responses, (31% response). Majority of respondents were satisfied with Overall Helpfulness (95%), Outcome of Service (96%), Ease of Access (93%), and Promptness of Service (90%). They also noted that their careers had at least slightly improved in the following areas: Mentorship (76%), Research Methods (75%), Skill Development (77%), Research Direction (71%) and Collaboration (80%). Furthermore, 96% responded positively to returning to TraCS. The feedback received was shared with service administrators and NC TraCS leadership to identify areas of improvement and further strengthen their services. Concerns, when present, were addressed by service directors or the overall PI’s. DISCUSSION/SIGNIFICANCE OF IMPACT: Need to communicate expectations to customers the expected turn-around time for help emerged as a clear take-away. In response, TraCS leadership is working to improve staffing and workflows for efficient service delivery including expectation management, especially among the most popular services.

OBJECTIVES/GOALS: The goals of this evaluation were 1) to describe the pilot grant application cycle and processes at NC TraCS, 2) to illustrate the impact of pilot grants on extramural grant funding, and 3) to provide a framework for other institutions to utilize for the evaluation of pilot grant programs. METHODS/STUDY POPULATION: From 2009-2019 the NC TraCS pilot program funded 925 projects, varying from $2,000 to $100,000. Pilot grants are available to any researcher affiliated with the university as well as partner institutions and community stakeholders. For this evaluation we analyzed data on pilot applicants (demographics, type of pilot, funding status, resubmissions, etc.) and outcomes (extramural funding, publications, etc.) yielded from funded pilots. In addition to summary statistics, we also calculated return on investment (ROI) for the program as a whole and by specific grant type. We will use bibliometric network analysis to assess productivity, citation impact, and scope of collaboration. RESULTS/ANTICIPATED RESULTS: There have been 2,777 submitted proposals with an acceptance rate of 33.3%. Unfunded proposals can resubmit, 61.8% of resubmitted applications are successfully funded, and 29.6% of funded applications are resubmissions. The $2,000 awards accounted for 43.4% of all grants awarded but only accounted for 6.4% of all pilot funds awarded. Success of proposals was proportional to the number of applications from each academic unit. 60.8% of funded applicants were affiliated with the School of Medicine and account for 65.3% of all funding awarded from 2009-2019. Additionally, we plan on analyzing return on investment rates to illustrate the impact of pilot awards on future research funding. DISCUSSION/SIGNIFICANCE OF IMPACT: Pilot grants can lead to subsequent extramural grants, publications, and successful translation of research into practice. This evaluation will assist our institution in understanding the impact of pilot grants and will provide a road map for other institutions evaluating their own programs.

The purpose of the article is to describe the progress of the Clinical and Translational Science Award (CTSA) Program to address the evaluation-related recommendations made by the 2013 Institute of Medicine’s review of the CTSA Program and guidelines published in CTS Journal the same year (Trochim et al., Clinical and Translational Science 2013; 6(4): 303–309). We utilize data from a 2018 national survey of evaluators administered to all 64 CTSA hubs and a content analysis of the role of evaluation in the CTSA Program Funding Opportunity Announcements to document progress. We present four new opportunities for further strengthening CTSA evaluation efforts: (1) continue to build the collaborative evaluation infrastructure at local and national levels; (2) make better use of existing data; (3) strengthen and augment the common metrics initiative; and (4) pursue internal and external opportunities to evaluate the CTSA program at the national level. This article will be of significant interest to the funders of the CTSA Program and the multiple stakeholders in the larger consortium and will promote dialog from the broad range of CTSA stakeholders about further strengthening the CTSA Program’s evaluation.

OBJECTIVES/SPECIFIC AIMS: The Wake Forest Clinical and Translational Science Institute (CTSI) put in place a KL2 program in 2007 and introduced an 18-month Translational Research Academy (TRA) in 2010. The TRA provides education and leadership development training, research support services, mentoring, and networking opportunities to 15-20 early-career clinical and translational researchers, including those receiving KL2 awards. The KL2 and TRA programs make up the Mentored-Career Development Core that is administered by the Wake Forest CTSA Education Program. Over the years, the program administrators have collected feedback from the graduates on what they liked and did not like about the programs. However, a comprehensive evaluation to understand the impact of the trainings on helping scholars advance their research and their research careers was not conducted. The purpose of this evaluation was to assess if and how the KL2 and TRA programs are helping scholars advance their research and career in research. METHODS/STUDY POPULATION: Semi-structured phone interviews were conducted with 11 selected CTSI scholars from both KL2 and TRA programs. The interviews focused on 4 key areas: expectations coming into the program, role the program played in their career development over time, what else could/should have program done to support them and their research, and which of the other CTSI services were valuable in their career development. RESULTS/ANTICIPATED RESULTS: Even though scholars joined the program as an early-career researcher, the amount of experience in research they had varied and so did their areas of interest. The participants in the program also included both clinician and basic researchers. Scholars came into the program with different level of expectations and drive to use this program as a platform to lift their research careers. The conversations during these interviews gave an insight on career trajectories that the scholars have taken before and after joining the academy. For example, among the scholars that had graduated from the programs, 5 of the 6 had received a career development award, all 6 had their own grants to support their research, 3 had received professional promotions, and all 6 have an administrative leadership role they play, in addition to focusing on their research career. The information on where the scholars were at the beginning and the pathway they have taken to get to where they are now allowed us to better understand what aspects of the program was most valuable. The scholars noted that sessions around grant writing and developing specific aims were very helpful. Among the services provided, having a grant editor support was something that everyone noted as the most important service to them, even after leaving the program. DISCUSSION/SIGNIFICANCE OF IMPACT: The impact of the interviews and candid feedback provided by the scholars have been immensely valuable to program administrators to really understand the impact the program (and which components specifically) is having for the scholars. Looking at the career trajectories of the scholars, it was noted that scholars coming in to the academy with some experience doing research and ability to continue engaging in research benefited the most from what the program had to offer. Scholars that were still very early in their research career gained knowledge but were not always able to apply it because they were not working towards a grant at the time. Using the data on career trajectories, recommendations for improving the program, and other CTSI services that were most valuable, the program administrators decided to redesign the curriculum. The new version of the program is now tailored for scholars who have research experience and are working towards a career development grant such as a K or R. This will allow them to have a curriculum that is more intense and hands-on with an expectation that the scholars will submit the application towards the end of the program. A separate program is being developed for early-career researchers who are still setting their foot into t field to provide them basic research competencies through ad hoc courses and seminars.

OBJECTIVES/SPECIFIC AIMS: The goal of the Wake Forest Clinical and Translational Science Institute (WF CTSI) Mentor Academy is to contribute to increasing the next generation of faculty with competencies specific to research mentoring. The curriculum of the Mentor Academy is adapted from an evidence-based national curriculum developed by the National Research Mentoring Network and includes 20 contact hours of didactic and experiential training, complemented with outside readings and assignments. A pre-post-follow-up competency assessment is built in as part of the curriculum for both participants and their current mentees. The purpose of this study was to assess self-rated research mentoring competencies among the Mentor Academy participants to better understand the effectiveness of the Mentor Academy. METHODS/STUDY POPULATION: A total of 37 mid-level or early senior faculty members from WF have participated in the 3 Mentor Academy cohorts that have completed so far. All of the participants receive 5% salary support and are expected to regularly participate in Mentor Academy sessions; complete a pre, post, and 6-month follow-up self-assessments; and provide a list of their active mentees. The identified mentees are also asked to assess the participating mentors’ research mentoring competencies before the start and 6-months after the end of the Mentor Academy. The same list of 26 mentoring competencies are included in the self-assessments for both mentors and mentees. RESULTS/ANTICIPATED RESULTS: The initial results of the self-assessments suggest that mentors are coming into the academy with a rather high self-assessed competency ratings. The change in competency ratings pre/post is not as significant. On average the change in self-assessed competency ratings increases by 1.0 on a 7-point scale. Interestingly enough, for 2 of the cohorts were mentees were also asked to assess their mentors’ competencies, the mentees rated their mentors as having a higher competency (for all 26 items) than what the mentor rated themselves, at both pre and 6-month follow-up assessments. DISCUSSION/SIGNIFICANCE OF IMPACT: After compiling data for 3 different cohorts, we are consistently seeing similar patterns in self-assessed competency ratings; participants are coming in with a high level of competency and an increased level of competency rating by mentees. These findings need to be further considered. For example, the program administrators need to discuss how participants are recruited, if we are recruiting the intended users, and what should we be expecting as an outcome(s) of the program. We also need to further explore different perceptions of mentor-mentee relationships and expectations to see how reliable are the data from mentees. A collaboration with the National Research Mentoring Network is also needed to see how the self-assessed competencies compare to those utilizing their curriculum outside of WF.