Recruitment of participants and their retention in randomized controlled trials (RCTs) is key for research efficiency. However, for many trials, recruiting and retaining participants meeting the eligible criteria is extremely challenging. Digital tools are increasingly being used to identify, recruit and retain participants. While these tools are being used, there is a lack of quality evidence to determine their value in trial recruitment.

The aim of the main study was to identify the benefits and characteristics of innovative digital recruitment and retention tools for more efficient conduct of RCTs. Here we report on the qualitative data collected on the characteristics of digital tools required by trialists, research participants, primary care staff, research funders and Clinical Trials Units (CTUs) to judge them useful. A purposive sampling strategy was used to identify 16 participants from five stakeholder groups. A theoretical framework was informed from results of a survey with UKCRC registered CTUs. Semi-structured interviews were conducted and analysed using an inductive approach. A content and thematic analysis was used to explore the stakeholder's viewpoint and the value of digital tools.

The content analysis revealed that ‘barriers / challenges ‘ and ‘awareness of evidence’ were the most commonly discussed areas. Three key emergent themes were present across all groups: ‘security and legitimacy of information’, ‘inclusivity’, and ‘availability of human interaction’. Other themes focused on the engagement of stakeholders in their use and adoption of digital technology to enhance the recruitment/retention process. We also noted some interesting similarities and differences between practitioner and participant groups.

The key emergent themes clearly demonstrate the use of digital technology in the recruitment and retention of participants in trials. The challenge, however, is using these existing tools without sufficient evidence to support the usefulness compared to traditional techniques. This raises important questions around the potential value for future research.

Recruitment of participants to, and their retention in, Randomized Controlled Trials (RCTs) is a key determinant of research efficiency, but is challenging. Digital tools and media are increasingly used to reduce costs, waste and delays in the conduct and delivery of research. The aim of this UK Clinical Trials Unit (CTU) survey was to identify which digital recruitment and retention tools are being used to support RCTs, their benefits and success characteristics.

A survey was sent to all UK Clinical Research Collaboration (UKCRC)-registered CTUs with a webinar to help increase completion. A logic model and definitions of a “digital tool” were developed by iterative refinement by project team members, the Advisory Board (NIHR Research Design service, NHS Trust, NIHR Clinical Research Networks and patient input) and CTUs.

A total of 24/52 (46%) CTUs responded, 6 (25%) of which stated no prior use. Database screening tools (e.g. CPRD, EMIS) were the tool most widely used (45%) for recruitment and were considered very effective (67%). The most mentioned success criteria were saving GP time and reaching more patients. Social media was second (27%), but estimated effectiveness varied considerably, with only 17% stating very effective. Fewer retention tools were used, with SMS / email reminders reported most (10/15 67%), but certainty about effectiveness varied. A detailed definition on what constitutes a digital tool with examples and a logic model showing relationships between the resources, activities, outputs and outcomes for digital tools was developed.

Database screening tools are the most commonly used digital tool for recruitment, with clear success criteria and certainty about effectiveness. Our detailed definition of what constitutes a digital tool, with examples, will inform the NIHR research community about choices and help them identify potential tools to support recruitment and retention.

New approaches are needed to safely reduce emergency admissions to hospital by targeting interventions effectively in primary care. A predictive risk stratification tool (PRISM) identifies each registered patient's risk of an emergency admission in the following year, allowing practitioners to identify and manage those at higher risk. We evaluated the introduction of PRISM in primary care in one area of the United Kingdom, assessing its impact on emergency admissions and other service use.

We conducted a randomized stepped wedge trial with cluster-defined control and intervention phases, and participant-level anonymized linked outcomes. PRISM was implemented in eleven primary care practice clusters (total thirty-two practices) over a year from March 2013. We analyzed routine linked data outcomes for 18 months.

We included outcomes for 230,099 registered patients, assigned to ranked risk groups.

Overall, the rate of emergency admissions was higher in the intervention phase than in the control phase: adjusted difference in number of emergency admissions per participant per year at risk, delta = .011 (95 percent Confidence Interval, CI .010, .013). Patients in the intervention phase spent more days in hospital per year: adjusted delta = .029 (95 percent CI .026, .031). Both effects were consistent across risk groups.

Primary care activity increased in the intervention phase overall delta = .011 (95 percent CI .007, .014), except for the two highest risk groups which showed a decrease in the number of days with recorded activity.

Introduction of a predictive risk model in primary care was associated with increased emergency episodes across the general practice population and at each risk level, in contrast to the intended purpose of the model. Future evaluation work could assess the impact of targeting of different services to patients across different levels of risk, rather than the current policy focus on those at highest risk.

Emergency admissions to hospital are a major financial burden on health services. In one area of the United Kingdom (UK), we evaluated a predictive risk stratification tool (PRISM) designed to support primary care practitioners to identify and manage patients at high risk of admission. We assessed the costs of implementing PRISM and its impact on health services costs. At the same time as the study, but independent of it, an incentive payment (‘QOF’) was introduced to encourage primary care practitioners to identify high risk patients and manage their care.

We conducted a randomized stepped wedge trial in thirty-two practices, with cluster-defined control and intervention phases, and participant-level anonymized linked outcomes. We analysed routine linked data on patient outcomes for 18 months (February 2013 – September 2014). We assigned standard unit costs in pound sterling to the resources utilized by each patient. Cost differences between the two study phases were used in conjunction with differences in the primary outcome (emergency admissions) to undertake a cost-effectiveness analysis.

We included outcomes for 230,099 registered patients. We estimated a PRISM implementation cost of GBP0.12 per patient per year.

Costs of emergency department attendances, outpatient visits, emergency and elective admissions to hospital, and general practice activity were higher per patient per year in the intervention phase than control phase (adjusted δ = GBP76, 95 percent Confidence Interval, CI GBP46, GBP106), an effect that was consistent and generally increased with risk level.

Despite low reported use of PRISM, it was associated with increased healthcare expenditure. This effect was unexpected and in the opposite direction to that intended. We cannot disentangle the effects of introducing the PRISM tool from those of imposing the QOF targets; however, since across the UK predictive risk stratification tools for emergency admissions have been introduced alongside incentives to focus on patients at risk, we believe that our findings are generalizable.

A predictive risk stratification tool (PRISM) to estimate a patient's risk of an emergency hospital admission in the following year was trialled in general practice in an area of the United Kingdom. PRISM's introduction coincided with a new incentive payment (‘QOF’) in the regional contract for family doctors to identify and manage the care of people at high risk of emergency hospital admission.

Alongside the trial, we carried out a complementary qualitative study of processes of change associated with PRISM's implementation. We aimed to describe how PRISM was understood, communicated, adopted, and used by practitioners, managers, local commissioners and policy makers. We gathered data through focus groups, interviews and questionnaires at three time points (baseline, mid-trial and end-trial). We analyzed data thematically, informed by Normalisation Process Theory (1).

All groups showed high awareness of PRISM, but raised concerns about whether it could identify patients not yet known, and about whether there were sufficient community-based services to respond to care needs identified. All practices reported using PRISM to fulfil their QOF targets, but after the QOF reporting period ended, only two practices continued to use it. Family doctors said PRISM changed their awareness of patients and focused them on targeting the highest-risk patients, though they were uncertain about the potential for positive impact on this group.

Though external factors supported its uptake in the short term, with a focus on the highest risk patients, PRISM did not become a sustained part of normal practice for primary care practitioners.