Hostname: page-component-848d4c4894-75dct Total loading time: 0 Render date: 2024-05-23T19:37:07.435Z Has data issue: false hasContentIssue false

Promoting the systematic use of real-world data and real-world evidence for digital health technologies across Europe: a consensus framework

Published online by Cambridge University Press:  14 September 2023

Divya Srivastava*
Department of Health Policy, London School of Economics and Political Science, London, UK
Cornelia Henschke
Deptartment of Health Care Management, Technische Universität Berlin, Berlin Centre for Health Economics Research (BerlinHECOR), Berlin, Germany
Lotta Virtanen
Welfare State Research and Reform Unit, Finnish Institute for Health and Welfare (THL), Helsinki, Finland
Eno-Martin Lotman
Cardiac Intensive Care Unit, North Estonia Medical Centre, Tallinn, Estonia
Rocco Friebel
Department of Health Policy, London School of Economics and Political Science, London, UK
Vittoria Ardito
Center for Research on Health and Social Care Management (CERGAS), SDA Bocconi School of Management, Milano, Italy
Francesco Petracca
Center for Research on Health and Social Care Management (CERGAS), SDA Bocconi School of Management, Milano, Italy
Corresponding author: Divya Srivastava; Email:
Rights & Permissions [Opens in a new window]


Despite the acceleration in the use of digital health technologies across different aspects of the healthcare system, the full potential of real-world data (RWD) and real-world evidence (RWE) arising from the technologies is not being utilised in decision-making. We examine current national efforts and future opportunities to systematically use RWD and RWE in decision-making in five countries (Estonia, Finland, Germany, Italy and the United Kingdom), and then develop a framework for promotion of the systematic use of RWD and RWE. A review assesses current national efforts, complemented with a three-round consensus-building exercise among an international group of experts (n1 = 44, n2 = 24, n3 = 24) to derive key principles. We find that Estonia and Finland have invested and developed digital health-related policies for several years; Germany and Italy are the more recent arrivals, while the United Kingdom falls somewhere in the middle. Opportunities to promote the systematic use of RWD and RWE were identified for each country. Eight building blocks principles were agreed through consensus, relating to policy scope, institutional role and data collection. Promoting post-market surveillance and digital health technology vigilance ought to rely on clarity in scope and data collection with consensus reached on eight principles to leverage RWD and RWE.

Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (, which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.
Copyright © The Author(s), 2023. Published by Cambridge University Press

1. Introduction

Post-market surveillance (PMS) systems are in place to collect relevant data in a systematic manner for medical devices and involve activities conducted by manufacturers once a product enters the market, to collect information on its quality, performance and safety (European Union, 2017). For regulators, the surveillance system is a tool to take action if there are safety concerns and risks of continued use of the medical device, which outweigh the benefits (WHO, 2021). This paper's objectives are (1) to investigate current efforts and future opportunities to systematically use real-world data (RWD) and real-world evidence (RWE) in PMS to support decision-making of digital health technologies (DHTs) and (2) to develop a framework to set out guiding principles (i.e. good practice guidelines) informed by the literature review and Delphi consensus-building exercise. Our research is timely because this is a new policy area where no country has yet established a well-designed, robust system, in part because digital health policy frameworks are a work in progress in many countries. In this paper, DHTs refer to health technologies as defined by the National Institute for Health and Care Excellence (NICE), including applications (apps), programmes and software used in the health and care system, which may be standalone or combined with other products such as medical devices or diagnostic tests (NICE, 2022). The definition is similar in scope to the one proposed by the World Health Organization in their guidelines on the use of digital health interventions. It refers to ‘a broad umbrella term encompassing eHealth, including mHealth or mobile wireless technologies for health, as well as emerging areas, such as the use of advanced computing sciences in ‘big data’, genomics and artificial intelligence’ (WHO, 2019).

A robust system of PMS has implications on the data needs for evidence generation and evidence thresholds, but also on how to learn from the technologies and inform policy once the DHT is on the market. In this paper, RWD and RWE are based on the US Food and Drugs Administration (FDA) definition (Box 1).

Box 1. RWD and RWE definitions

There are untapped benefits from large volumes of data being created. Data could inform providers and patients, feed back into decision-making similar to the health technology assessment (HTA) to inform regulation or set evidence thresholds. It could monitor the technology once it is on the market and could be used to inform upstream design of the DHT, or to understand the degree of generalisability of the DHT. However, it remains unknown whether digital technologies/consultations uncover more need, hence raising the burden. There could be implications on workload for general practitioners with a relatively young/healthy population that could be serviced quickly via online consultation (telemedicine), leading to equity concerns and commissioning/payment considerations. Online consultations attract younger, healthier patients while those who are older/or with long-term conditions are likely to be less comfortable attending remote consultations (Salisbury et al., Reference Salisbury, Quigley, Hex and Aznar2020; Sounderajah et al., Reference Sounderajah, Clarke, Yalamanchili, Acharya, Markar, Ashrafian and Darzi2021; Wieringa et al., Reference Wieringa, Neves, Rushforth, Ladds, Husain, Finlay, Pope and Greenhalgh2022). For example, informing policy could systematically embed user feedback from providers and patients and on how to better mitigate the digital divide (e.g. disparities in the use of technologies among underserved populations). Unlike DHTs, there are ‘pharmacovigilance’ policies and systems for pharmaceutical products that are partly harmonised in Europe for the monitoring and evaluation of medicines once they are on the market, which also helps to understand their effects on different patient profiles and patient sub-groups.

One of the current challenges is the sheer volume of data arising from DHTs. Data come from a variety of sources, with no current international framework on guiding principles for data access and analytics, creating by default, global databases (e.g. an app may contain data from different countries). Related to large volumes of data is how to use data and to support interoperability at various levels (e.g. DHTs, healthcare settings, national borders) as raised in the EU4Digital's eHealth Network on patients' rights in cross-border healthcare. The initiative aims to connect national authorities responsible for eHealth issues to help share policy surrounding eHealth interoperability and standardisation (European Union, 2021). The extent to which real-time data are accessible to support analytics or evidence generation remains an open question.

This paper investigates the opportunities for systematically embedding RWD and RWE in PMS to support decision-making of DHTs drawing on five country experiences (Estonia, Finland, Germany, Italy and United Kingdom with a focus on England). Decision-making in this paper considers decisions taken at different levels in the system, such as by patients, healthcare workers or national and regional institutions (e.g. payers or HTA bodies). We then develop a framework including guiding principles informed by the literature review and Delphi exercise, through an international consensus-building exercise. The consensus framework will bring benefits in decision-making to support nationally and internationally coherent decision-making in PMS. It will assist health authorities in setting out clarity in policy scope, role of institutions in collection of RWD, regulators to set requirements using RWE and digital health developers in their compliance.

Our research explores policy issues around RWD and RWE for DHTs to support their PMS and we propose ‘digital health technology vigilance’ similar in concept to pharmacovigilance to avert potential adverse effects; we argue that this is currently missing in country policies and will be a key aspect for DHTs in the future (CIOMS, 2022). Country policies that include DHT vigilance in a systematic way in their surveillance will improve our understanding of the risks and functionalities of DHTs. A robust PMS system with information on RWD will facilitate the benefits for decision-making including monitoring for quality, safety and adverse events to inform users, governments and digital health developers.

2. Methods

2.1 Literature review

A review of the literature was conducted, inspired by the scoping review method, to map policy issues related to RWD and RWE for DHTs. This included a review of relevant literature, including grey literature in five select countries: Estonia, Finland, Germany, Italy and the United Kingdom (UK).

Countries were selected based on the extent of the application of RWD and RWE in the health system (Table 1). The countries are not a representative sample but rather our interest was to reflect a mix of models of healthcare delivery, including different levels of decentralisation, financing healthcare digitalisation, adoption of digital technologies and population perceptions in regard to collection, use and sharing of data (Ferré et al., Reference Ferré, de Belvis, Valerio, Longhi, Lazzari, Fattore, Ricciardi and Maresso2014; Habicht et al., Reference Habicht, Reinap, Kasekamp, Sikkut, Aaben and van Ginneken2018; Keskimäki et al., Reference Keskimäki, Tynkkynen, Reissell, Koivusalo, Syrjä, Vuorenkoski, Rechel and Karanikolos2019; Vehko et al., Reference Vehko, Ruotsalainen and Hyppönen2019; Blümel et al., Reference Blümel, Spranger, Achstetter, Maresso and Busse2020; OECD et al., 2021; Anderson et al., Reference Anderson, Pitchforth, Edwards, Alderwick, McGuire and Mossialos2022).

Table 1. Country characteristics: Estonia, Finland, Germany, Italy and the United Kingdom (England)

The literature search focussed on the following aspects: the countries' existing national frameworks and policy objectives, national efforts, ongoing activities and opportunities to further embed RWD and RWE into decision-making. These aspects were applied to each country to guide the search and gain information of the national context. Some countries have had policies in place for several years, but the study focussed on the past decade (2012–2022) to capture relevant and recent developments in digital health policy. An important contribution of our approach was that beyond publications in English, the literature search was conducted in the country's native language (Estonian, Finnish, German, Italian), across relevant national databases and sources of grey literature to discover relevant legal documents, reports and research articles (Table 2).

Table 2. Literature search for review

2.2 2.2 International consensus-building exercise

The Delphi was informed by findings from the review. In the literature, we identified what was missing from current country practices related to measuring and monitoring of DHTs and what potential the countries have to make measuring and monitoring more systematic. We formed initial guiding principles based on our interpretation of the reviewed documents and then these guiding principles were further developed with the expert panel in the Delphi consensus exercise.

The second part of the methodology employed a consensus-building exercise using the Delphi method. The purpose of Delphi was to develop a framework based on guiding principles to embed RWD and RWE systematically to support decision-making for policy makers, providers and patients. The Delphi rounds involved a range of international experts identified through academic, clinical and research networks.

2.3 Expert recruitment and consent

Experts in DHTs, including experts and researchers affiliated with national and regional institutions, payers, healthcare professionals and experts from patient representative organisations were identified in each of the review countries. To achieve a broad range of views, and country representation beyond our country focus, members of the European Health Policy Group (EHPG) were also invited to participate in the wider clinical, academic and practitioner community. The EHPG consists of a wide range of European experts with affiliations in health policy, research and clinical settings. Experts were contacted via the academic institutions of the study authors. A total of 496 experts were contacted.

2.4 Ethical approval

This study was approved by the London School of Economics and Political Science (approval number 101617). The study's pre-registration can be accessed on the Open Science Framework ( Participation in the Delphi study was voluntary and all participants provided informed consent to participate via email.

2.5 Data collection

The Delphi method was employed through online questionnaires over three rounds between 30 September 2022 and 15 November 2022. The study team agreed to construct the questionnaire in English. In each round, the Delphi questionnaire had two sections. The first part contained building block principles as key elements of a digital health policy framework. The second part contained principles that related to leveraging RWD and RWE for PMS and DHT vigilance.

The experts were asked to rate the importance for each principle using a five-point Likert scale: 1 = exclude, 2 = low priority, 3 = medium priority, 4 = high priority, 5 = essential to include. Additionally, the questionnaire collected demographic information on participants (geographic location, age, gender and occupation).

In the first round Delphi, consenting experts accessed the background information of the study via email, which included a link to the online questionnaire on Qualtrics. Experts were asked to rate and comment on the proposed principles. The given response time was 14 days and two reminders were sent before the first round closed.

Based on the comments from the first round Delphi, the study team modified the principles and established new principles for the subsequent two rounds. In the second and third round, all experts who had provided their consent were invited with their unique personal electronic link to access the questionnaire via Welphi (an online survey platform specifically designed for Delphi consensus-building exercises). Experts were asked to rate and comment on the proposed principles. Experts received a summary of anonymised ratings and comments from each of the previous rounds. This information was provided so experts could consider whether they wished to rank items differently. Experts were given 8 days to complete the second round and 5 days to complete the third round. In both rounds, two reminders were sent before the round closed.

2.6 Analysis

The analysis assessed the distribution of responses to determine the pooled level of agreement for each principle. Consensus was reached for a principle when at minimum 75% of the experts had given a rating of at least four, indicating that the inclusion of the principle in the proposed framework was agreed as a high priority. Guidelines for the Delphi survey technique recommend defining consensus as 75% is acceptable level of agreement (Hasson et al., Reference Hasson, Keeney and McKenna2000). In the literature, some studies have defined consensus as ranging between 75 and 80% (Hasson et al., Reference Hasson, Keeney and McKenna2000; Keeney et al., Reference Keeney, McKenna and Hasson2011; Jünger et al., Reference Jünger, Payne, Brine, Radbruch and Brearley2017). Thereafter, sensitivity analysis was conducted. The analysis was informed by all comments received alongside the distribution of responses including the mean, median, interquartile range (IQR), standard deviation and variance. Analysis was conducted using Excel.

3. Results

3.1 Literature review

The country experiences highlight examples of policies to guide decision-making, but the depth and scope of these policies varies (see Appendix 1 for country summaries). Estonia and Finland have invested and developed digital health-related policies for several years since the 1990s. Germany and Italy are the most recent arrivals, in terms of significant innovations regarding digital health policies, within the past three years. The United Kingdom falls somewhere in the middle.

National efforts have focussed on a range of different activities including reimbursement, data collection efforts and monitoring of quality. Estonia set out reimbursement requirements for some DHTs. Finland's well-established infrastructure supports the flow of digital-related data to national repositories which collect some aggregate information on DHT usage. Germany recently developed an explicit approach for fast-track approvals of digital health applications. Italy set out a long-term vision to collect and harness large amounts of RWD including some digital health-related data on telemedicine. England in the United Kingdom systematically collects aggregate RWD on primary care visits (face-to-face and virtual) to support outcomes, commissioning and regulate quality (Table 3).

Table 3. Summary of findings from Estonia, Finland, Germany, Italy and the United Kingdom (England)

Although the pandemic catalysed a shift in demand towards the uptake of DHTs offering potential sources of RWD and RWE, there are opportunities to further embed and develop approaches for using RWD and RWE systematically. Sources of RWD could bring further insights on patient experience and support clinical decision-making as seen in the example of remote monitoring of patients with cardiac-related conditions in Estonia. National aggregate data collections do not yet offer ways to distinguish the type of DHT, nor offer systematic information on the uptake among underserved patient groups, but in Finland, the existing nationwide health data resources could provide better opportunities for them. The example from Italy suggests that large amounts of RWD require alignment with data protection but also conditions to create an open and accepting environment to foster expertise in DHTs. The examples of Germany and the UK stress the need for ongoing data collection to inform the product's benefits, quality, safety and reimbursement once the DHT is available on the market.

Country experiences with RWD and RWE arising from DHTs are in their infancy. These examples highlight opportunities and challenges that have implications for post-marketing surveillance but also for a notion we term as DHT vigilance. Adopting a total product lifecycle (TPLC) approach may better align the RWD and RWE data needs and requirements for decision-makers in policy and regulatory circles, manufacturers, providers and patients because it adopts a wholistic approach in the product's lifecycle from design to PMS (Figure 1). The Delphi consensus exercise builds on these findings, with guiding principles that could inform a country's digital health policy framework.

Figure 1. Different evidence needs arising from RWD and RWE.

Source: CDRH Transparency: Total Product Life Cycle (TPLC) | FDA.

3.1.1 Results of the Delphi consensus-building exercise

Of the invited experts, 44 completed round one (response rate 9%). Of them, 24 (67%) completed round two, and similarly, 24 (67%) completed round three. Table 4 shows the demographic characteristics of experts in each round. Close to half or more were aged between 40 and 59 years, with a fairly even split between male and female experts in the first two rounds. Most participating experts came from the countries focussed on in this study: Finland, Germany, Italy, United Kingdom and Estonia. The experts came from a range of settings with most coming from scientist/academic research, healthcare professionals and national/regional institutions.

Table 4. Demographics of experts

Summary of main findings. A total of 16 out of 20 principles reached consensus. Consensus was reached on five out of 15 principles in the first round. Feedback from round one informed the re-wording of principles and three additional principles in the Delphi questionnaire. In the second round Delphi, consensus was reached on a further eight out of 16 principles. Feedback from round two led to re-wording principles and removing one principle. Consensus was reached on three out of seven principles in the third round (see Appendix 2). Consensus ranged from 75 to 87%.

There are eight building block principles for a DHT framework presented in decreasing order of level of agreement (Table 5). They set out a range of considerations on explicit definitions of DHTs, institutional requirements and a strategy around the collection and use of RWD and RWE including for when a DHT is on the market (see Appendix 2).

Table 5. Consensus reached on building block principles

Eight principles are proposed to set out how to consider where RWD and RWE could inform decision-making with respect to PMS and DHT vigilance presented in decreasing order of level of agreement (Table 6).

Table 6. Consensus reached on leveraging RWD and RWE for post-market surveillance and digital health technology vigilance

In summary, the first set of guiding principles identified areas to inform the development and digital health policy framework. The second set of principles extends these notions with respect to leveraging RWD and RWE once the DHT is on the market. The intersection between the two brings to the fore their interdependence with respect to PMS and a concept we propose as DHT vigilance. This paper argues that taking a TPLC approach is essential to frame these considerations.

The agreed consensus speaks to the need for key principles to set out a clear strategy, clarity around the application and use of RWD and RWE to support an agile environment drawing on data collections and evidence that are manageable and appropriate to inform decision-making. The agreement on guiding principles reflects their importance across a cross-section of key stakeholders concerning RWD and RWE. This is an important finding on its own that reflects a convergence in thinking among a group of stakeholders with a range of expertise.

Sensitivity analysis. We conducted sensitivity analysis on our findings (see Appendix 2). The threshold level of agreement was adjusted by five percentage points downwards and upwards to test the robustness of the results. With a 70% level of agreement (instead of 75%), the main findings stayed the same; the remaining four that were not included were still below 70% (between 56 and 65%). With an 80% level of agreement (instead of 75%), eight principles were removed: three building block principles and five from the PMS and DHT vigilance principles. The remaining principles with 80% agreement still reflect the priorities in terms of policy scope, institutional role and data collection requirements.

Our intention was to reach consensus among experts from five main affiliations: national/regional institution, payer, healthcare professional, scientist/academic research, patient/patient organisation. The experts contacted in the study countries, and via the EHPG and in academic institutions reflect this wide mix. A breakdown of responses by affiliations with sufficient sample size (five or more) was available for three out of the five categories. The sensitivity analysis shows that there is broad consensus across most of the guiding principles (Appendix 2).

4. Limitations

It is important to note the limitations in our analysis. Our review focussed on the most relevant evidence and was not systematic in design, so the results of the review are not exhaustive. However, the review was guided by the study team's pre-existing expertise on the subject, which promoted the construction of an overall picture of the national activities obtained with the chosen search strategy. The Delphi study was conducted over a month and so there may be key experts in the five countries that may have been missed. Nevertheless, every effort was made to gather a cross-section of views from the stakeholder groups in these country settings. Contact with all the country experts including introductory emails were sent, if necessary, in their native languages to describe their role and expectations. Experts who agreed to participate were informed that the survey and instructions were administered in English. There may be differences in interpretation of the statements by the experts, but we do not expect that differences in interpretation of the statements will have a large impact on our findings. When looking across affiliations, there is broad consensus with most of the guiding principles which further supports our study's results.

5. Discussion and conclusion

This paper set out to understand how RWD and RWE arising from DHTs are used in decision-making. The findings from the review of five countries filled this knowledge gap, identifying opportunities for each country and their progress with respect to digital health policymaking. The review informed the development of guiding principles to embed RWD and RWE systematically for DHTs to inform decision-making. The proposed principles were reached using the Delphi consensus-building technique among an international group of experts in the field. Consensus was reached for 16 out of 20 principles, which represent building blocks related to policy scope, institutional role and data collection and principles to promote PMS and DHT vigilance (Figures 2 and 3). For the four where consensus was not reached, the distribution of responses, however, reflects their relative high importance (responses had a median value of four) (Appendix 2). Currently, the principles are ranked in order of agreement. A next step for research will be to consider reflecting their importance and relationship with respect to the principles of economics and role and motivation for government intervention in the market of DHTs.

Figure 2. Building block principles to support a total product lifecycle approach.

Figure 3. Principles for post-market surveillance and digital health technology vigilance.

In conclusion, the three main contributions of this paper include first a more nuanced understanding of RWD and RWE with respect to DHTs in the countries reviewed. Second, this is the first study to offer the development of guiding principles to improve digital health policymaking for RWD and RWE introducing the notion of DHT vigilance. Third, the findings from the Delphi exercise highlight the need for greater focus on RWD and RWE across Europe. Considerations include clear criteria for decision-making, institutional involvement in overseeing how and where RWD and RWE are appropriate for decision-making. International involvement and engagement would be useful given the large amounts of data being collected, along with flexibility around evidence needs for the most appropriate study designs and a digital health strategy that is agile given the fast-paced nature of this sector.

Supplementary material

The supplementary material for this article can be found at


The authors would like to express their gratitude to all participants of the Delphi consensus-building exercise. The authors acknowledge the support of Oriana Ciani and Francesco Longo in earlier stages of the paper's research.

Financial support

L. V. received funding from the Strategic Research Council at the Academy of Finland (grants 327145 and 352501 for the DigiIN Project), without any influence on the design of the study, collection, analysis and interpretation of the data, or the conclusions.


Agenzia per l'Italia Digitale (2014) Linee guida per la presentazione dei piani di progetto regionali per il FSE. Available at Scholar
Ahlqvist, J and Kalliola, M (2021) How can digital therapeutics help Europe? (Working Paper), Sitra.Google Scholar
Ahlqvist, J and Kalliola, M (2022) Digitaaliset terapiat – Vaikuttavuutta uudistuviin terveyspalveluihin [Digital therapeutics – Effectiveness for renewable health services] (Working Paper), Sitra.Google Scholar
Anderson, M, Pitchforth, E, Edwards, N, Alderwick, H, McGuire, A and Mossialos, E (2022) The United Kingdom: health system review. Health Systems in Transition (Vol. 24(1).Google Scholar
Atherton, H, Brant, H, Ziebland, S, Bikker, A, Campbell, J, Gibson, A, McKinstry, B, Porqueddu, T and Salisbury, C (2018) The potential of alternatives to face-to-face consultation in general practice, and the impact on different patient groups: a mixed-methods case study. Health Services and Delivery Research 6, 1199.CrossRefGoogle Scholar
BfArM (2022) The fast-track process for digital health applications (DiGA) according to Section 139e SGB V. Available at Scholar
Blümel, M, Spranger, A, Achstetter, K, Maresso, A and Busse, R (2020). Germany: health system review. Health Systems in Transition (Vol. 22(5)).Google Scholar
BMG (2019 a) Gesetzes für mehr Sicherheit in der Arzneimittelversorgung (GSAV). Available at Scholar
BMG (2019 b) Digitale-Versorgung-Gesetz (DVG). Bundesgesundheitsministerium. Available at Scholar
BMG (2020) Krankenhauszukunftsgesetz (KHZG) – Bundesgesundheitsministerium. Available at Scholar
BMG (2023) Gemeinsam digital. Digitalisierungsstrategie für das Gesundheitswesen und die Pflege. Available at Scholar
CIOMS (2022) Science and activities relating to the detection, assessment, understanding and prevention of adverse effects or any other drug-related problem. Available at,of%20adverse%20effects%20or%20any%20other%20drug-related%20problemGoogle Scholar
Cylus, J, Richardson, E, Findley, L, Longley, M, O'Neill, C and Steel, D (2015) United Kingdom: health system review. Health Systems in Transition. WHO Regional Office for Europe on behalf of the European Observatory on Health Systems and Policies.Google Scholar
Dickstein, K, Normand, C, Auricchio, A, Bogale, N, Cleland, JG, Gitt, AK, Stellbrink, C, Anker, SD, Filippatos, G, Gasparini, M, Hindricks, G, Blomström Lundqvist, C, Ponikowski, P, Ruschitzka, F, Botto, GL, Bulava, A, Duray, G, Israel, C, Leclercq, C and Linde, C (2018). CRT Survey II: a European Society of Cardiology survey of cardiac resynchronisation therapy in 11 088 patients – who is doing what to whom and how? European Journal of Heart Failure 20, 10391051.CrossRefGoogle Scholar
Essén, A, Stern, AD, Haase, CB, Car, J, Greaves, F, Paparova, D, Vandeput, S, Wehrens, R and Bates, DW (2022) Health app policy: international comparison of nine countries’ approaches. NPJ Digital Medicine 5, 31.CrossRefGoogle ScholarPubMed
Estonian Health Insurance Fund. (2020) Digitaalsete tervisetehnoloogiate hindamisraamistik.Google Scholar
Estonian Health Insurance Fund. (2022) Available at Scholar
European Commission (2020) Digital Economy and Society Index DESI. Digital public services. Available at Scholar
Article 83, Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (2017).Google Scholar
European Union (2021) EU4Digital. Available at Scholar
Ferré, F, de Belvis, AG, Valerio, L, Longhi, S, Lazzari, A, Fattore, G, Ricciardi, W and Maresso, A. (2014) Italy: Health System Review. Health Systems in Transition (Vol. 16(4)).Google Scholar
Habicht, T, Reinap, M, Kasekamp, K, Sikkut, R, Aaben, L and van Ginneken, E (2018) Estonia: Health system review. Health Systems in Transition (Vol. 20(1)).Google Scholar
Häkkinen, P, Mölläri, K, Saukkonen, S-M, Väyrynen, R, Mielikäinen, L and Järvelin, J (2019) Hilmo - Sosiaali- ja terveydenhuollon hoitoilmoitus 2020: Määrittelyt ja ohjeistus [National Care Notification Register of health and social care 2020: definitions and guidelines]. Available at Scholar
Hammersley, V, Donaghy, E, Parker, R, McNeilly, H, Atherton, H, Bikker, A, Campbell, J and McKinstry, B (2019) Comparing the content and quality of video, telephone, and face-to-face consultations: a non-randomised, quasi-experimental, exploratory study in UK primary care. British Journal of General Practice 69, e595.CrossRefGoogle ScholarPubMed
Hasson, F, Keeney, S and McKenna, H (2000) Research guidelines for the Delphi survey technique. Journal of Advanced Nursing 32, 10081015.CrossRefGoogle ScholarPubMed
Haverinen, J, Keränen, N, Falkenbach, P, Maijala, A, Kolehmainen, T and Reponen, J (2019) Digi-HTA: health technology assessment framework for digital healthcare services. Finnish Journal of eHealth and eWelfare, 11, 326341.CrossRefGoogle Scholar
Heponiemi, T, Jormanainen, V, Leemann, L, Manderbacka, K, Aalto, A-M and Hyppönen, H (2020) Digital divide in perceived benefits of online health care and social welfare services: national cross-sectional survey study. Journal of Medical Internet Research 22, e17616.CrossRefGoogle ScholarPubMed
Hughes, G, Wood, G and Shaw, S (2020) Video consultations in the NHS: summary findings from a UK-wide survey.Google Scholar
Hyppönen, H and Aalto, AM (2019) Citizen experiences of e-health and e-welfare services. In Vehko, T, Ruotsalainen, S and Hyppönen, H (eds), E-health and e-welfare of Finland: Check Point 2018. Helsinki: Finnish Institute for Health and Welfare (THL), pp. 148167. Available at Scholar
Jonsson, PM, Pikkujämsä, S and Heiliö, P-L (2019) National quality registers in healthcare and social services – operating model, organization and financing (Report 16/2019, Issue. Available at Scholar
Jünger, S, Payne, SA, Brine, J, Radbruch, L and Brearley, SG (2017) Guidance on conducting and REporting DElphi studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliative Medicine 31, 684706.CrossRefGoogle ScholarPubMed
Kaihlanen, A-M, Virtanen, L, Buchert, U, Safarov, N, Valkonen, P, Hietapakka, L, Hörhammer, I, Kujala, S, Kouvonen, A and Heponiemi, T (2022) Towards digital health equity – a qualitative study of the challenges experienced by vulnerable groups in using digital health services in the COVID-19 era. BMC Health Services Research 22, 188.CrossRefGoogle ScholarPubMed
Keeney, S, McKenna, H and Hasson, F (2011) The Delphi Technique in Nursing and Health Research. Chichester: Wiley. Available at Scholar
Keskimäki, I, Tynkkynen, L-K, Reissell, E, Koivusalo, M, Syrjä, V, Vuorenkoski, L, Rechel, B and Karanikolos, M (2019) Finland: health system review.Google Scholar
Koivisto, J (2021) Esiselvitys sosiaali- ja terveydenhuollon kansallisten digitalisaatio-ohjelmien arviointikehikon kehittämiseksi [Preliminary study to develop an evaluation framework for national digitalisation programs in health and social care] (Working paper 28/2021, Issue. Available at Scholar
Kyytsönen, M, Aalto, A-M and Vehko, T (2021 a) Sosiaali- ja terveydenhuollon sähköinen asiointi 2020–2021: Väestön kokemukset [Social and health care online service use in 2020–2021: Experiences of the population] (Report 7/2021). Available at Scholar
Kyytsönen, M, Vehko, T, Jormanainen, V, Aalto, A-M and Mölläri, K (2021 b) Terveydenhuollon etäasioinnin trendit vuosien 2013-2020 Avohilmon aineistossa [Trends in remote healthcare in the Avohilmo data in 2013-2020]. Available at Scholar
Laki julkisista hankinnoista ja käyttöoikeussopimuksista [Act on Public Procurement and Concession] 1397/2016 (2016) Available at Scholar
Laki sosiaali- ja terveydenhuollon asiakastietojen sähköisestä käsittelystä [Act on the Electronic Processing of Client Data in Healthcare and Social Welfare] 784/2021 (2021) Available at Scholar
Laki sosiaali- ja terveystietojen toissijaisesta käytöstä [Act on the Secondary Use of Health and Social Data] 552/2019 (2019 a) Available at Scholar
Laki Terveyden ja hyvinvoinnin laitoksesta annetun lain muuttamisesta [Act amending the Act on the Finnish Institute for Health and Welfare] 553/2019 (2019 b) Available at Scholar
Lantzsch, H, Eckhardt, H, Campione, A, Busse, R and Henschke, C (2022) Digital health applications and the fast-track pathway to public health coverage in Germany: Challenges and opportunities based on first results. Available at Scholar
Lotman, EM and Viigimaa, M (2020) Digital health in cardiology: the Estonian perspective. Cardiology 145, 2126.CrossRefGoogle ScholarPubMed
Ministero della Salute (2011) The National eHealth Strategy. National context, state of implementation and best practices. Available at Scholar
Ministero della Salute (2016 a) Patto per la sanità digitale. Available at Scholar
Ministero della Salute (2016 b) Piano Nazionale della Cronicità. Available at Scholar
Ministri, PDCD (2014) Intesa tra il Governo, le Regioni e le Province autonome di Trento e Bolzano sul documento recante ‘Telemedicina – Linee di indirizzo nazionali’. Available at Scholar
Ministry of Finance (2022) General government fiscal plan for 2023–2026. Available at Scholar
NHS England and NHS Improvement (2021) Supporting general practice in 2021/22. Available at Scholar
NHSX (2019 a) Artificial intelligence how to get it right.Google Scholar
NHSX (2019 b) NHSX: new joint organisation for digital, data and technology. Available at Scholar
NICE (2022) Evidence Standards Framework for Digital Health Technologies. NICE.Google Scholar
Northern Ostrobothnia Hospital, D (2022) Digi-HTA: assessment method. Available at Scholar
Nuffield Trust (2020) Quality Watch. Digital and remote care in the NHS during covid-19.Google Scholar
OECD (2019) UK Country Health Profile.Google Scholar
OECD, European Observatory on Health Systems and Policies (2021). Italy: Country Health Profile 2021. Available at Scholar
Oliver, D. (2019). David Oliver: lessons from the Babylon health saga. BMJ, 365, l2387. ScholarPubMed
Petracca, F, Ciani, O, Cucciniello, M and Tarricone, R (2020) Harnessing digital health technologies during and after the COVID-19 pandemic: context matters. Journal of Medical Internet Research 22, e21815.CrossRefGoogle ScholarPubMed
Presidenza del Consiglio dei Ministri (2021) Piano Nazionale di Ripresa e Resilienza. Available at Scholar
Primary Care Training Centre (2020) The changing world of the asthma telephone review. Available at Scholar
Reponen, J, Keränen, N, Ruotanen, R, Tuovinen, T, Haverinen, J and Kangas, M (2021) Tieto- ja viestintäteknologian käyttö terveydenhuollossa vuonna 2020: Tilanne ja kehityksen suunta [Use of information and communications technology in Finnish health care in 2020. Current situation and trends]. Available at Scholar
Ricciardi, W and Tarricone, R (2021) The evolution of the Italian National Health Service. The Lancet 398, 21932206.CrossRefGoogle ScholarPubMed
Salisbury, C, Quigley, A, Hex, N and Aznar, C (2020) Private video consultation services and the future of primary care. Journal of Medical Internet Research 22, e19415.CrossRefGoogle ScholarPubMed
Scobie, S and Castle-Clarke, S (2019) What can the NHS learn from learning health systems? Available at Scholar
Sheikh, A, Anderson, M, Albala, S, Casadei, B, Franklin, BD, Richards, M, Taylor, D, Tibble, H and Mossialos, E (2021) Health information technology and digital innovation for national learning health and care systems. The Lancet Digital Health 3, e383e396.CrossRefGoogle ScholarPubMed
Simovic, S, Providencia, R, Barra, S, Kircanski, B, Guerra, JM, Conte, G, Duncker, D, Marijon, E, Anic, A and Boveda, S (2022) The use of remote monitoring of cardiac implantable devices during the COVID-19 pandemic: an EHRA physician survey. EP Europace 24, 473480.CrossRefGoogle ScholarPubMed
Skrami, E, Carle, F, Villani, S, Borrelli, P, Zambon, A, Corrao, G, Trerotoli, P, Guardabasso, V and Gesuita, R (2020) Availability of real-world data in Italy: a tool to navigate regional healthcare utilization databases. International Journal of Environmental Research and Public Health 17, 8.CrossRefGoogle Scholar
Sounderajah, V, Clarke, J, Yalamanchili, S, Acharya, A, Markar, SR, Ashrafian, H and Darzi, A (2021) A national survey assessing public readiness for digital health strategies against COVID-19 within the United Kingdom. Scientific Reports, 11, 5958.CrossRefGoogle ScholarPubMed
Srivastava, D, Scarbrough, H and Stavropoulou, C (2020) Current regulatory challenges to support the spread of digital health technologies, Centre for Healthcare Innovation Research Policy Report, London: City University.Google Scholar
Stern, AD, Brönneke, J, Debatin, JF, Hagen, J, Matthies, H, Patel, S, Clay, I, Eskofier, B, Herr, A, Hoeller, K, Jaksa, A, Kramer, DB, Kyhlstedt, M, Lofgren, KT, Mahendraratnam, N, Muehlan, H, Reif, S, Riedemann, L and Goldsack, JC (2022) Advancing digital health applications: priorities for innovation in real-world evidence generation. The Lancet Digital Health 4, e200e206.CrossRefGoogle ScholarPubMed
TK (2022) Vier Forderungen zur Optimierung von Apps auf Rezept. Availabe at Scholar
Torkki, P and Mäki-Opas, T (2021) Vaikuttavuuden vuosikymmentä rakentamassa [Building a decade of effectiveness] (Statement by University of Helsinki and University of Eastern Finland, Issue.Google Scholar
Valtioneuvoston asetus (582/2017) erikoissairaanhoidon työnjaosta ja eräiden tehtävien keskittämisestä [Government Decree on the division of labour in specialised care and the centralisation of certain tasks 582/2017] (2017) Available at Scholar
Vehko, T, Ruotsalainen, S and Hyppönen, H (2019) E-health and e-welfare of Finland: Check Point 2018. Available at Scholar
WHO (2019) Recommendations on digital health interventions.Google Scholar
WHO (2021) Guidance for post-market surveillance and market surveillance of medical devices, including in vitro diagnostics.Google Scholar
Wieringa, S, Neves, AL, Rushforth, A, Ladds, E, Husain, L, Finlay, T, Pope, C and Greenhalgh, T (2022) Safety implications of remote assessments for suspected COVID-19: qualitative study in UK primary care. BMJ Quality & Safety. doi: 10.1136/bmjqs-2021-013305CrossRefGoogle ScholarPubMed
Figure 0

Figure 1

Table 1. Country characteristics: Estonia, Finland, Germany, Italy and the United Kingdom (England)

Figure 2

Table 2. Literature search for review

Figure 3

Table 3. Summary of findings from Estonia, Finland, Germany, Italy and the United Kingdom (England)

Figure 4

Figure 1. Different evidence needs arising from RWD and RWE.Source: CDRH Transparency: Total Product Life Cycle (TPLC) | FDA.

Figure 5

Table 4. Demographics of experts

Figure 6

Table 5. Consensus reached on building block principles

Figure 7

Table 6. Consensus reached on leveraging RWD and RWE for post-market surveillance and digital health technology vigilance

Figure 8

Figure 2. Building block principles to support a total product lifecycle approach.

Figure 9

Figure 3. Principles for post-market surveillance and digital health technology vigilance.

Supplementary material: PDF

Srivastava et al. supplementary material

Appendices 1 and 2

Download Srivastava et al. supplementary material(PDF)
PDF 233 KB