Whenever access to a promising health technology is being considered (e.g., taking a decision on reimbursement or coverage), decision makers often face a lack of strong evidence not only on the technology's efficacy and safety, but also on its effectiveness and cost-effectiveness (Reference LeLorier10). A comprehensive HTA is difficult due to lack of information. Several countries have therefore developed policy frameworks to allow access to the technology on the condition that additional evidence is generated through: “access with evidence generation (AEG)” mechanisms (Reference Hutton, Trueman and Henshall4;Reference Laupacis, Paterson and Mamdani9). These mechanisms differ according to country.
The European network for Health Technology Assessment (EUnetHTA) project (Reference Kristensen, Lampe and Chase6;Reference Kristensen, Mäkelä and Allgurin Neikter7) offered us an opportunity to construct a model policy framework for AEG mechanisms (Reference Carbonneil, Quentin and Lee-Robin1). The framework comprises five steps (Reference Carbonneil, Quentin and Lee-Robin1): (i) a first HTA report pinpoints lacking evidence and proposes a plan for data collection; (ii) a decision is made on conditional and temporary access to the technology based on this report and is accompanied by a request for evidence generation; (iii) an interim period of conditional access to the technology follows, during which the requested data are collected; (iv) a second HTA report is produced that includes the additional evidence generated; (v) a revised decision is then made on access to the technology based on this second report.
A revised decision may nevertheless be difficult to make due to the many barriers to evidence generation (Reference Carbonneil, Quentin and Lee-Robin1;Reference Chalkidou, Hoy and Littlejohns2;Reference Hutton, Trueman and Henshall4;Reference Tunis and Pearson12) (Box 1). An important barrier at the international level is the lack of structured collaboration among the HTA agencies involved in AEG mechanisms. Information is passed on, mostly by email, from person to person or within informal networks (e.g., INAHTA listserv). This is inefficient, time-consuming, a source of misunderstanding, and does not permit easy data storage and sharing. In addition, the information passed on is often incomplete or inadequate. More importantly, there is no way to ensure that the work is not duplicated, wasting valuable time and resources.
Box 1. Main Barriers to Evidence Generation
A driving force in the EUnetHTA Project has been the development of communication facilities to support collaborative work among EUnetHTA Partners (3). Hence, the Project has offered an ideal opportunity to establish structured forms of collaboration on evidence generation relating to promising health technologies.
OBJECTIVES
One of the objectives of Work Package 7 (WP7) of the EUnetHTA project has been to determine the types of structured collaboration that would facilitate evidence generation and to create a Web-based toolkit that would support this collaboration.
METHODS
The work schedule included (i) identification of needs, (ii) definition of Web site content, (iii) technical development (electronic forms and online database) guaranteeing interoperability with the EUnetHTA HTA Information system, (iv) Web site testing by the WP7-A Lead Partner. WP7 Partners defined different modes of collaboration to facilitate evidence generation on promising health technologies and decided to focus first on sharing information. They discussed and agreed upon a list of relevant information to be shared and developed standard forms for information sharing in a three-step process: (i) planning technical development, (ii) developing and reviewing the first draft of the forms, (iii) amending the forms.
Two pilot tests of the forms were carried out. In the first pilot test, three WP7 Partners completed the forms designed for requesting information, using as examples technologies in which they were particularly interested. All WP7 Partners were then asked to complete the forms designed to answer queries (posting information form). Their comments were used to amend the forms. In the second pilot test, all WP7 Partners were asked to test the forms for requesting and posting information on technologies in which they were particularly interested. Their comments were again used to amend the forms. Each completed form was checked to ensure that the information provided was in line with the items on the forms.
A Web-based toolkit (a Web site) with a database was developed for information sharing based on the use of the standard forms that were developed.
RESULTS
Types of Collaboration
Three levels of cooperation between EUnetHTA Partners on promising health technologies were defined: sharing information, coordinated action, and joint action (Box 2).
Box 2. Levels of Collaboration
Information to be Shared
The following key information to be shared was selected (Box 3).
Box 3. Information on a Promising Technology
Design and Pilot Testing of the Standard Forms
Draft standard data forms for requesting and supplying information were tested by seven of the thirty-one WP7 Partners in the first pilot test and by six in the second pilot test. Participation rate was low. Most participants were either WP7 Partners with substantial experience of AEG mechanisms or the contrary, Partners with little experience. Experienced Partners were able to consolidate the quality of their work and less experienced Partners were able to learn.
The participating partners tested the forms for twenty-one technologies (Table 1). Information was requested on thirteen technologies; only six of thirteen requests received a reply. Information on eight technologies was provided spontaneously. More than one request or reply was recorded for four technologies (bevacizumab in age-related macular degeneration, transient elastography, implantable cardioverter defibrillator, intensity-modulated radiation therapy [IMRT]).
Table 1. Lists of Health Technologies Used in the Pilot Tests
aFirst pilot test.
bSecond pilot test.
During quality control of the requests and replies, each participant was contacted at least once for details. In general, the participant had not understood one or more items. These were reworded for greater clarity.
Example of the Value of Sharing Information
In the pilot test, a request for information was completed by a Partner who was planning to reassess IMRT. The request was for information on coverage, effectiveness, and appropriateness of use. Two Partners replied. However, only one provided valuable information on the status of IMRT in their country (marketing authorization, i.e., CE marking and coverage), on the AEG mechanisms that had been set up (registry and monitoring of use), the protocol implemented, and the sources of registry funding. In the light of this reply, the Partner making the request decided to postpone the reassessment of IMRT until the additional data being collected by the Partner who replied would become available. The new data would be included in the reassessment report and used to support the decision on coverage.
Creation of the Web-based Toolkit: A Web Site
The structured standard forms for information entry are available on a Web site (Eunethta Interface to Facilitate Furthering of Evidence Level (http://eiffel.eunethta.has-sante.fr/)). This Web site is for use by EUnetHTA Partners only and can be accessed through a link from the EUnetHTA Web site.
Web Site Content. The Web site provides access to the forms for requesting information (request form), posting information in response to a request (posting form), and posting information spontaneously (spontaneous posting form). The Web site also provides an online queryable database containing all the information requested or posted. It will be fed automatically, as and when the forms are filled. The forms completed for twenty-one technologies during the pilot testing have been entered into the database.
When completing the forms, users must specify if the information provided is confidential (to be sent only to the user requesting the information) or semiconfidential (available to database users, i.e., EUnetHTA Partners).
Each Partner is responsible for the quality of the information they provide. Moreover, this information undergoes a quality control to ensure that the information entered corresponds to the items on the form. This quality control will be performed by a data manager of the HTA team in charge of the Web site.
Web Site Access. Figure 1 shows how the Web site is used and how information is processed: The user searches the database for information on a promising health technology (Action 1). If no or insufficient information is retrieved, the user completes the standard request form on the request page (Action 2). The request form undergoes quality control (Action 3). The request form is published on the Web site (Action 4), and all EUnetHTA Partners are notified by email. Partners who can provide the information requested complete the standard posting form (Action 5). The posting form undergoes quality control (Action 6).The posting form is then published on the Web site (Action 7), and the user who requested the information is informed by email that a member has responded to the request. Any user can provide information spontaneously by completing the spontaneous posting form (Action 5’). All information exchanged is automatically stored in the database. All members are informed of entries by email alert.
Figure 1. How to use the Web site.
Intended Web Site Users. The intended Web site users are EUnetHTA Partners, that is, “publicly funded” organizations that produce or contribute to HTA. Three user profiles were identified (Box 4).
Box 4. Web Site User Profiles
Users can use results obtained elsewhere when they are applicable to their local context.
DISCUSSION
Providing timely access to promising technologies is a major issue in healthcare. Several European countries have developed access with evidence generation (AEG) mechanisms to make decisions on access, but have found that information is scarce, not easy to find, and that evidence is difficult to generate (Reference Carbonneil, Quentin and Lee-Robin1).
WP7 Partners developed standardized forms, which replace informal emails, for requesting and supplying information on promising health technologies. These forms are available on a dedicated Web site for sharing information on evidence generation among EUnetHTA Partners. The Web site is called Eunethta Interface to Facilitate Furthering of Evidence Level (http://eiffel.eunethta.has-sante.fr/). By this means, transfer of information becomes more efficient and the information garnered is more comprehensive. More importantly, the process permits easy storage of information, saves time, and can ultimately avoid duplication of work.
The overview of AEG mechanisms conducted by WP7-A found that the amount of required evidence generated for access to a promising technology differed according to country (Reference Carbonneil, Quentin and Lee-Robin1). Implementation of the AEG mechanism could be full (all the evidence required was generated), partial (only some of the evidence required was generated), or passive (none of the additional evidence required was generated). In practice, few countries obtain all the evidence they need for a sound and robust decision. The Web site will help them attain a critical mass of evidence faster, for a more evidence-based decision.
We identified three potential obstacles in using the Web site: (i) the “not invented here” syndrome, (ii) frustration, and (iii) habit.
(i) “Not invented here” syndrome: Users may be reluctant to use information that does not come from their own AEG mechanism as they cannot control its quality directly. A way to overcome this obstacle is to issue regular reminders to users that they must ensure the accuracy of the information they supply. The supplier is responsible for the quality of the information given. In addition, before making use of the information, the interested party can directly email the supplier to obtain confirmation that the information is indeed still accurate.
(ii) Frustration: Clearly, users will be frustrated if the information they need is not in the database, as its content will not immediately reach a critical mass. To speed up supply, EUnetHTA Partners will be regularly solicited for information. Users may also be frustrated, even annoyed, if the information is obsolete. Hence, users will also be asked regularly to update information.
(iii) Habit: Users may be reluctant to request and/or post information through the Web site and will just continue to share information by means of informal emails. This obstacle is already known to the International Network of Agencies for Health Technology Assessment (INAHTA) which has long-term experience of database management (HTA database, Briefs, Checklists and Impact forms). INAHTA thus decided to increase the frequency of reminders to encourage information supply and to send regularly e-mail alerts about new additions to database content (INAHTA Secretariat, personal communication). The further development of our Web site will take their experience into account and include a training session on Web site use for EUnetHTA Partners.
Three limitations of a more general nature were also identified: (i) transferability of the information, (ii) lack of transparency, and (iii) wording.
(i) Transferability of information: Can the information actually be transferred directly to be shared? Differences among countries, for example, differences in terminology, technology use, physician training, and population risks, come into play (Reference Rosten, Chase and Hicks11;Reference Turner, Chase and Milne13). Users will need to use the domain classification of the HTA Core Model (e.g., description and technical characteristics, current use) (Reference Lampe, Mäkelä and Velasco-Garrido8), the glossary of HTA terms: INAHTA (5), adaptation terms (Reference Rosten, Chase and Hicks11), and the toolkit for adapting an HTA report to their local context (Reference Turner, Chase and Milne13).
(ii) Lack of transparency: Only WP7 Partners were involved in this work. Moreover, they were involved in testing the forms for information requests and supply only, with a rather disappointing participation rate. They were not involved in Web site testing. Transparency will increase when we have developed tests of the Web site for all EUnetHTA Partners. Web site access is currently restricted to EUnetHTA Partners because some of the data on promising technologies (e.g., clinical data) are confidential and not intended for the general public. However, access to the nonconfidential items may be provided in the future (e.g., level of diffusion of the health technology in different healthcare systems, status of HTA reports).
(iii) Wording: The wording used in the forms needs to be improved. During the pilot tests, explanations had to be given to each participating Partner on how to complete the forms. The terms new and promising also need to be defined according to the level of diffusion of the technology in the healthcare system. For example, some Partners considered technologies such as implantable cardiac defibrillators and tympanostomy tubes to be promising, whereas they are in routine use in other countries. We plan to develop an online glossary of key terms used on the Web site to facilitate common understanding.
For the Web site to become fully operational, it will be necessary to include the user reminders identified above concerning information supply, quality, and updating, to provide a glossary of key terms, to perform large-scale tests involving all EUnetHTA Partners, and to organize training sessions on the final product. The development of a Web site for sharing information meets the needs of the first of the three levels of collaboration we defined for facilitating evidence generation on promising health technologies. They correspond to the three levels of collaboration for the EUnetHTA Collaboration (voluntary information sharing, coordination of common activities, and joint actions) (3).
When no or insufficient evidence on promising technologies is available to share, it will be necessary to coordinate and even arrange joint actions to generate additional evidence. Toolkits need to be developed for these two higher levels of commitment:
(i) The toolkit to facilitate coordinated, but independent, generation of new evidence on a promising health technology may include a rapid agreement process to define core protocols for collecting a common set of data and methodological developments on core protocols (e.g., type of data needed to fill the evidence gaps, study/register designs for monitoring studies).
(ii) The toolkit to facilitate joint actions may include a framework for collaborative collection of lacking evidence across EUnetHTA Partners and methodological developments on criteria for selecting the technologies to be monitored.
CONCLUSIONS
The Web site serves as a means to share information on evidence generation on promising health technologies. It should help countries reach robust decisions on the diffusion of these technologies and thus foster their timely adoption. However, the Web site will only be worthwhile if all EUnetHTA Partners agree to supply relevant, accurate, and updated information – and use it regularly. Committed Partners will need to oversee the operation and continuing development of the Web site.
CONTACT INFORMATION
Fabienne Quentin, PhD (f.quentin@has-sante.fr), Project Manager, Cédric Carbonneil, PhD (c.carbonneil@has-sante.fr), Project Manager, Céline Moty-Monnereau, MD, PhD (celine.moty-monnereau@sante.gouv.fr), Project Manager, Medical and Surgical Procedures Assessment, French National Authority for Health, 2 avenue du Stade de France, Saint-Denis La Plaine CEDEX F-93218, France
Elena Berti, MD (eberti@regione.emilia-romagna.it), Assistant, Clinical Governance Area, Agenzia Sanitaria e Sociale Regione Emilia-Romagna, 21 Via Aldo Moro, Bologna, Italy, 40127
Wim Goettsch, PhD (wgoettsch@cvz.nl), Consultant Pharmacoeconomics, Health Care Insurance Board, Eekholt 4, Diemen, The Netherlands, 1112XH
Sun Hae Lee-Robin, MD, MPH (sh.leerobin@has-sante.fr), Head of Department, Medical and Surgical Procedures Assessment, French National Authority for Health, 2 avenue du Stade de France, Saint-Denis La Plaine CEDEX, F-93218, France
