To send this article to your account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account.
Find out more about sending content to .
To send this article to your Kindle, first ensure firstname.lastname@example.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle.
Find out more about sending to your Kindle.
Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.
Background: Ethics has been part of health technology assessment (HTA) from its beginning in the 1970s, and is currently part of HTA definitions. Several methods in ethics have been used in HTA. Some approaches have been developed especially for HTA, such as the Socratic approach, which has been used for a wide range of health technologies. The Socratic approach is used in several ways, and there is a need for harmonization to promote its usability and the transferability of its results. Accordingly, the objective of this study was to stimulate experts in ethics and HTA to revise the Socratic approach.
Methods: Based on the current literature and experiences in applying methods in ethics, a panel of ethics experts involved in HTA critically analyzed the limitations of the Socratic approach during a face-to-face workshop. On the basis of this analysis a revision of the Socratic approach was agreed on after deliberation in several rounds through e-mail correspondence.
Results: Several limitations with the Socratic approach are identified and addressed in the revised version which consists of a procedure of six steps, 7 main questions and thirty-three explanatory and guiding questions. The revised approach has a broader scope and provides more guidance than its predecessor. Methods for information retrieval have been elaborated.
Conclusion: The presented revision of the Socratic approach is the result of a joint effort of experts in the field of ethics and HTA. Consensus is reached in the expert panel on an approach that is considered to be more clear, comprehensive, and applicable for addressing ethical issues in HTA.
Background: Non small cell lung cancer (NSCLC) diagnosis and treatment is a highly complex process, requiring managerial skills merged with clinical knowledge and experience. Integrated care pathways (ICPs) might be a good strategy to overview and improve patient's management. The aim of this study was to review the ICPs of NSCLC patients in a University Hospital and to identify areas of quality improvement.
Materials and Methods: The electronic medical records of 169 NSCLC patients visited at the University Hospital were retrospectively reviewed. Quality of care (QoC) has been measured trough fifteen indicators, selected according main international Guidelines and approved by the multi-disciplinary team for thoracic malignancies. Results have been compared with those of a similar retrospective study conducted at the same hospital in 2008.
Results: A total of 146 patients were considered eligible. Eight of fifteen indicators were not in line with the benchmarks. We compared the results obtained in the two separate periods. Moreover, we process some proposal to be discussed with the general management of the hospital, aimed to redesign NSCLC care pathways.
Conclusions: ICPs confirm to be feasible and to be an effective tool in real life. The periodic measurement of QoC indicators is necessary to ensure clinical governance of patients pathways.
Objectives: The headroom method was introduced for the very early evaluation of the potential value of new technologies. It allows for establishing a ceiling price for technologies to still be cost-effective by combining the maximum effect a technology might yield, the maximum willingness-to-pay (WTP) for this effect, and potential downstream expenses and savings. Although the headroom method is QALY-based, not all innovations are expected to result in QALY gain.
Methods: This study explores the feasibility and usefulness of the headroom method in the evaluation of technologies that are unlikely to result in QALY gain. This will be illustrated with the diagnostic trajectory of complex pediatric neurology (CPN).
Results: Our headroom analysis showed a large room for improvement in the current diagnostic trajectory of CPN in terms of diagnostic yield. Combining this with a maximum WTP value for an additional diagnosis and the potential downstream expenses and savings, resulted in a total headroom of €15,028. This indicates that a new technology in this particular diagnostic trajectory, might be cost-effective as long as its costs do not exceed €15,028.
Conclusions: The headroom method seems a useful tool in the very early evaluation of medical technologies, also in cases when immediate QALY gain is unlikely. It allows for allocating healthcare resources to those technologies that are most promising. It should be kept in mind, however, that the headroom assumes an optimistic scenario, and for that reason cannot guarantee future cost-effectiveness. It might be most useful for ruling out those technologies that are unlikely to be cost-effective.
Objectives: The aim of this study is to report on the experiences, benefits, and challenges of patient and public involvement and engagement (PPIE) from a publicly funded early awareness and alert (EAA) system in the United Kingdom.
Methods: Using email, telephone, a Web site portal, Twitter and focus groups, patients and the public were involved and engaged in the recognized stages of an EAA system: identification, filtration, prioritization, early assessment, and dissemination.
Results: Approaches for PPIE were successfully integrated into all aspects of the National Institute for Health Research Horizon Scanning Research and Intelligence Centre's EAA system. Input into identification activities was not as beneficial as involvement in prioritization and early assessment. Patients gave useful insight into the Centre's Web site and engaging patients using Twitter has enabled the Centre to disseminate outputs to a wider audience.
Conclusions: EAA systems should consider involving and engaging with patients and the public in identification, prioritization, and assessment of emerging health technologies where practicable. Further research is required to examine the value and impact of PPIE in EAA activities and in the early development of health technologies.
Objectives: In many economic evaluations and reimbursement decisions, quality-adjusted life-years (QALYs) are used as a measure of benefit to assess effectiveness of novel therapies, often based on the EQ-5D 3-level questionnaire. As only five dimensions of physical and mental well-being are reflected in this tool, significant aspects of the patient experience may be missed. We evaluate the use of the EQ-5D as a measurement of clinical change across a wide range of disorders from dermatological (acne) to life-threatening (metastatic cancers).
Methods: We analyze published studies on the psychometric properties of the EQ-5D 3-level questionnaire, extracting information on the Visual Analogue Scale versus Index score, Standardized Response Mean, and Effect Size. These are compared with ranges generally accepted to represent good responsiveness in the psychometric literature.
Results: We find that only approximately one in five study populations report subjective health state valuation of patients within 5 percent of the score attributed by the EQ-5D index, and more than 40 percent of studies report unacceptable ceiling effects. In the majority of studies, responsiveness of the EQ-5D index was found to be poor to moderate, based on Effect Size (63 percent poor–moderate) and Standardized Response Mean (72 percent poor–moderate).
Conclusions: We conclude that the EQ-5D index does not adequately reflect patient health status across a range of conditions, and it is likely that a significant proportion of the subjective patient experience is not accounted for by the index. This has implications for economic evaluations of novel drugs based on evidence generated with the EQ-5D.
Objectives: Zika virus (ZikaV) is currently one of the most important emerging viruses in the world which has caused outbreaks and epidemics and has also been associated with severe clinical manifestations and congenital malformations. Traditional approaches to combat the ZikaV outbreak are not effective for detection and control. The aim of this study is to propose a cloud-based system to prevent and control the spread of Zika virus disease using integration of mobile phones and Internet of Things (IoT).
Methods: A Naive Bayesian Network (NBN) is used to diagnose the possibly infected users, and Google Maps Web service is used to provide the geographic positioning system (GPS)-based risk assessment to prevent the outbreak. It is used to represent each ZikaV infected user, mosquito-dense sites, and breeding sites on the Google map that helps the government healthcare authorities to control such risk-prone areas effectively and efficiently.
Results: The performance and accuracy of the proposed system are evaluated using dataset for 2 million users. Our system provides high accuracy for initial diagnosis of different users according to their symptoms and appropriate GPS-based risk assessment.
Conclusions: The cloud-based proposed system contributed to the accurate NBN-based classification of infected users and accurate identification of risk-prone areas using Google Maps.
Objectives: The aim of this study was to review 5 years of activity from a new system devised by the National Institute for Health and Care Excellence (NICE), for assessing medical devices and diagnostics aimed at identifying and speeding adoption of technologies with clinical and cost advantages, compared with current practice in the United Kingdom healthcare system.
Methods: All eligible notified technologies were classified using the Food and Drug Administration and Global Medical Device Nomenclature nomenclatures. Decisions about selecting technologies for full assessment to produce NICE recommendations were reviewed, along with the reasons given to companies for not selecting products.
Results: Between 2009 and 2014, 186 technologies were notified (46 percent therapeutic and 54 percent diagnostic). Thirty-nine were judged ineligible (no regulatory approval), and 147 were considered by an independent committee. Of these, eighty (54 percent) were not selected for full assessment, most commonly because of insufficient evidence (86 percent): there were uncertainties specifically about benefits to the health service (54 percent), to patients (39 percent), and about cost (24 percent). The remaining 67 were selected and assessed for Medical Technology guidance (52 percent) (noninferior and/or lower cost consequences than current practice), for Diagnostics guidance (43 percent) or other NICE recommendations about adoption and use. Classifying technologies by two different systems showed no selection bias for any technology type or disease area.
Conclusions: Identifying new or under-used devices and diagnostics with potential benefits and promoting their adoption is important to health services in the United Kingdom and worldwide. This new system offers a means of fostering both uptake and further research. Lack of research data on new products is a major obstacle to evaluation.
Background: Increasingly, healthcare decision makers demand quality evidence in a short timeframe to support urgent and emergent decisions related to procurement, clinical practice, and policy. Health technology assessment (HTA) producers are responding by developing innovative approaches to evidence synthesis that can be executed more quickly than traditional systematic review. These approaches, and the broader implications they bring to bear on health decision making and policy development, however, are generally neither well-understood nor well-described. This study intends to contribute to an emerging literature around methodological approaches to rapid review in HTA by outlining those developed and implemented by the Canadian Agency for Drugs and Technologies in Health (CADTH).
Methods: Since 2005, CADTH has developed and implemented a rapid review approach that synthesizes evidence to support informed healthcare decisions and policy. Rapid Response reports are tailored to the identified needs of Canadian health decision makers, representing a range of options with regard to depth, breadth, and time-to-delivery.
Results: Preliminary observations indicate that CADTH's approach to rapid evidence review is generally well-received by Canadian health decision makers; real-world case studies provide pragmatic examples of how health decision makers have used Rapid Response reports to support evidence-informed health decisions across Canada.
Conclusions: Rapid review is becoming an increasingly important approach to evidence synthesis, both within and external to the field of HTA. Transparent reporting of the methods used to develop rapid review products will be critical to the assessment of their relevance, utility and effects in a range of contexts.
Objectives: Procedures and new medical devices are typically introduced into healthcare systems with limited evidence, when they might be ineffective or unsafe. Systematic data collection (“registers”) can provide valuable “real world” evidence, but difficulties in funding registers are a major obstacle. A good economic case for the value of registers would therefore be useful.
Methods: (i) Literature search on specific purposes of registers. (ii) Surveys (a) of senior clinicians involved with registers, seeking examples of beneficial outcomes, and (b) of administrators, regarding costs of running registers. (iii) A scoping exercise for possible methods to value (financially) the outputs of registers.
Results: Four main categories of beneficial outcomes from registers were identified. These were—safety and quality assurance; training and quality improvement; complementing trial evidence and reducing uncertainty; and supporting trial research. Explicit examples of all these are presented, together with information about the costs of registers. Combining these with the scoping exercise we present suggestions for a methodology of assessing the value of registers across each of the categories.
Conclusions: This study is unique in addressing methods for determining the financial value of registers, based on the amount they cost versus the financial benefits which may result from the evidence generated. Developing the suggested methods could support the case for funding new registers, by showing that their use can benefit healthcare systems through more efficient use of resources, so justifying their costs.
Objectives: Reporting bias potentially threatens the validity of results in health technology assessment (HTA) reports. Our study aimed to explore policies and practices of HTA agencies regarding strategies to include previously unpublished data in their assessments, focusing on requests to industry for unpublished data.
Methods: We included international HTA agencies with publicly available methods papers as well as HTA reports. From the methods papers and recent reports we extracted information on requests to industry and on searches in trial registries, regulatory authority Web sites and for conference abstracts.
Results: Eighteen HTA agencies and seventy-three reports were included. Agencies’ methods papers showed variability regarding requests to industry (requests are routinely carried out in seven cases, not mentioned in six, at the discretion of HTA authors in three, and based on manufacturer applications in two), which were reflected in the reports investigated. As reporting of requests was limited, it often remained unclear whether unpublished data had been received. Searches in trial registries, at regulatory authorities or for conference abstracts are described as a routine or optional part of the search strategy in the methods papers of 9, 11, and 8 included agencies, respectively. A total of 52 percent, 39 percent, and 16 percent of reports described searches in trial registries, at regulatory agencies, and hand searching of conference proceedings.
Conclusion: International HTA agencies currently differ considerably in their efforts to address the issue of unpublished data. Requests to industry may constitute one strategy to access and include unpublished data, while agencies can learn from each other concerning successful practice.
Objectives: When incorporating treatment effect estimates derived from a random-effect meta-analysis it is tempting to use the confidence bounds to determine the potential range of treatment effect. However, prediction intervals reflect the potential effect of a technology rather than the more narrowly defined average treatment effect. Using a case study of robot-assisted radical prostatectomy, this study investigates the impact on a cost-utility analysis of using clinical effectiveness derived from random-effects meta-analyses presented as confidence bounds and prediction intervals, respectively.
Methods: To determine the cost-utility of robot-assisted prostatectomy, an economic model was developed. The clinical effectiveness of robot-assisted surgery compared with open and conventional laparoscopic surgery was estimated using meta-analysis of peer-reviewed publications. Assuming treatment effect would vary across studies due to both sampling variability and differences between surgical teams, random-effects meta-analysis was used to pool effect estimates.
Results: Using the confidence bounds approach the mean and median ICER was €24,193 and €26,731/QALY (95%CI: €13,752 to €68,861/QALY), respectively. The prediction interval approach produced an equivalent mean and median ICER of €26,920 and €26,643/QALY (95%CI: -€135,244 to €239,166/QALY), respectively. Using prediction intervals, there is a probability of 0.042 that robot-assisted surgery will result in a net reduction in QALYs.
Conclusions: Using prediction intervals rather than confidence bounds does not affect the point estimate of the treatment effect. In meta-analyses with significant heterogeneity, the use of prediction intervals will produce wider ranges of treatment effect, and hence result in greater uncertainty, but a better reflection of the effect of the technology.
Objectives: There is little specific guidance on performing an early cost-effectiveness analysis (CEA) of medical tests. We developed a framework with general steps and applied it to two cases.
Methods: Step 1 is to narrow down the scope of analysis by defining the test's application, target population, outcome measures, and investigating current test strategies and test strategies if the new test were available. Step 2 is to collect evidence on the current test strategy. Step 3 is to develop a conceptual model of the current and new test strategies. Step 4 is to conduct the early-CEA by evaluating the potential (cost-)effectiveness of the new test in clinical practice. Step 5 involves a decision about the further development of the test.
Results: The first case illustrated the impact of varying the test performance on the headroom (maximum possible price) of an add-on test for patients with an intermediate-risk of having rheumatoid arthritis. Analyses showed that the headroom is particularly dependent on test performance. The second case estimated the minimum performance of a confirmatory imaging test to predict individual stroke risk. Different combinations of sensitivity and specificity were found to be cost-effective; if these combinations are attainable, the medical test developer can feel more confident about the value of further development of the test.
Conclusions: A well-designed early-CEA methodology can improve the ability to develop (cost-)effective medical tests in an efficient manner. Early-CEAs should continuously integrate insights and evidence that arise through feedback, which may convince developers to return to earlier steps.
Objectives: Developing a search strategy for use in a systematic review is a time-consuming process requiring construction of detailed search strings using complicated syntax, followed by iterative fine-tuning and trial-and-error testing of these strings in online biomedical search engines.
Methods: Building upon limitations of existing online-only search builders, a user-friendly computer-based tool was created to expedite search strategy development as part of production of a systematic review.
Results: Search Builder 1.0 is a Microsoft Excel®-based tool that automatically assembles search strategy text strings for PubMed (www.pubmed.com) and Embase (www.embase.com), based on a list of user-defined search terms and preferences. With the click of a button, Search Builder 1.0 automatically populates the syntax needed for functional search strings, and copies the string to the clipboard for pasting into Pubmed or Embase. The offline file-based interface of Search Builder 1.0 also allows for searches to be easily shared and saved for future reference.
Conclusions: This novel, user-friendly tool can save considerable time and streamline a cumbersome step in the systematic review process.
Objectives: A rapid scoping review was performed to support the development of a new clinical technology platform. An iterative sifting approach was adopted to address the challenges posed by the nature of the review question and the extremely large volume of search results to be sifted within the timescales of the review.
Methods: This study describes the iterative sifting approach applied in the scoping review and a preliminary validation of the methods applied.
Results: The searches performed for the rapid scoping review retrieved 27,198 records. This was the full set of records subjected to the staged, iterative sifting approach and the subsequent validation process. The iterative sifting approach involved the screening for relevance of 17,354 (i.e., 63.8 percent) of the 27,198 records. A list of fifty-three potential biomarker names was generated as a result of this iterative sifting method, of which nineteen were selected by clinical specialists for further scrutiny. The preliminary validation involved the exhaustive sifting of the remaining 9,844 previously unsifted records. The validation process identified sixteen additional potential biomarker names not identified by the iterative sifting process. The clinical specialists subsequently concluded that none were of further clinical interest.
Conclusions: This study describes an approach to the screening of search records that can be successfully applied in appropriate review and decision problems to allow the prioritization of the most relevant search records and achieve time savings. Following further refinement and standardization, this iterative sifting method may have potential for further applications in reviews and other decision problems.
Objectives: The regularly structured adaptation of health technology assessment (HTA) programs is of utmost importance to sustain the relevance of the products for stakeholders and to justify investment of scarce financial resources. This study describes internal adjustments and external measures taken to ensure the Horizon Scanning Programme in Oncology (HSO) is current.
Methods: Formal evaluation methods comprising a survey, a download, an environmental analysis, and a Web site questionnaire were used to evaluate user satisfaction.
Results: The evaluation showed that users were satisfied with HSO outputs in terms of timeliness, topics selected, and depth of information provided. Discussion of these findings with an expert panel led to changes such as an improved dissemination strategy and the introduction of an additional output, that is, the publication of a league table of emerging oncology drugs. The rather high level of international usage and the environmental analysis highlighted a considerable overlap in topics assessed and, thus, the potential for international collaboration. As a consequence, thirteen reports were jointly published based on eleven “calls for collaboration.” To further facilitate collaboration and the usability of reports for other agencies, HSO reports will be adjusted according to tools developed at a European level.
Conclusions: Evaluation of the impact of HTA programs allows the tailoring of outputs to fit the needs of the target population. However, within a fast developing HTA community, estimates of impact will increasingly be determined by international collaborative efforts. Refined methods and a broader definition of impact are needed to ultimately capture the efficiency of national HTA programs.
Objectives: The objective of this study was to compare evidence requirements for health technology assessment of pharmaceuticals by national agencies across Europe responsible for reimbursement decisions focusing specifically on relative effectiveness assessment.
Methods: Evidence requirements from thirty-three European countries were requested and twenty-nine national agencies provided documents to review. Data were extracted from national documents (manufacturer's submission templates and associated guidance) into a purpose-made framework with categories covering information about the health condition, the technology, clinical effectiveness and safety.
Results: The level of detail in the required evidence varies considerably across countries. Some countries include specific questions while others request information under general headings. Some countries include all information in a single document, which may or may not include guidance on how to complete the template. Others have specific guidance documents or methods and process manuals that help with the completion of the submission templates. Despite differences in quantity and detail, the content of the evidence requirements is broadly similar. All countries ask for information on the health technology, target disease, and clinical effectiveness and safety. However, one country only requests clinical effectiveness information as part of cost-effectiveness analyses. We found twenty-six evidence requirements for which generic answers may apply across borders and nineteen in which countries requested nationally specific information.
Conclusions: This work suggests that it would be possible to put together a minimum set of evidence requirements for HTA to support reimbursement decisions across Europe which could facilitate collaboration between jurisdictions.
Objectives: The aim of this study was to determine the aspects of expert advice that decision makers find most useful in the development of evidence-based guidance and to identify the characteristics of experts providing the most useful advice.
Methods: First, semi-structured interviews were conducted with seventeen members of the Interventional Procedures Advisory Committee of the UK's National Institute of Health and Care Excellence. Interviews examined the usefulness of expert advice during guidance development. Transcripts were analyzed inductively to identify themes. Second, data were extracted from 211 experts’ questionnaires for forty-one consecutive procedures. Usefulness of advice was scored using an index developed through the qualitative work. Associations between usefulness score and characteristics of the expert advisor were investigated using univariate and multivariate analyses.
Results: Expert opinion was seen as a valued complement to empirical evidence, providing context and tacit knowledge unavailable in published literature, but helpful for interpreting it. Interviewees also valued advice on the training and experience required to perform a procedure, on patient selection criteria and the place of a procedure within a clinical management pathway. Limitations of bias in expert opinion were widely acknowledged and skepticism expressed regarding the anecdotal nature of advice on safety or efficacy outcomes. Quantitative analysis demonstrated that the most useful advice was given by clinical experts with direct personal experience of the procedure, particularly research experience.
Conclusions: Evidence-based guidance production is often characterized as a rational, pipeline process. This ignores the valuable role that expert opinion plays in guidance development, complementing and supporting the interpretation of empirical data.
Objectives: The aim of this study was to explore stakeholders’ points of views regarding the applicability and relevance of a framework for user involvement in health technology assessment (HTA) at the local level. We tested this framework in the context of the assessment of alternative measures to restraint and seclusion among hospitalized adults and those living in long-term-care facilities.
Methods: Twenty stakeholders (health managers, user representatives, and clinicians) from seven regions of Quebec participated in a semi-structured interview. A thematic analysis of the transcribed interviews was performed.
Results: The findings highlighted the relevance and applicability of the framework to this specific HTA. According to interviewees, direct participation of users in the HTA process allows them to be part of the decision-making process. User consultation makes it possible to consider the views of a wide variety of people, such as marginalized and vulnerable groups, who do not necessarily meet the requirements for participating in HTA committees. However, some user representatives emphasized that user consultation should be integrated into a more holistic and participatory perspective. The most frequent barrier associated with user involvement in HTA was the top-down health system, which takes little account of the user's perspective.
Conclusions: The proposed framework was seen as a reference tool for making practitioners and health managers aware of the different mechanisms of user involvement in HTA and providing a structured way to classify and describe strategies. However, there is a need for more concrete instruments to guide practice and support decision making on specific strategies for user involvement in HTA at the local level.
Objectives: This study assesses the use of routinely collected claims data for managed entry agreements (MEA) in the illustrative context of German statutory health insurance (SHI) funds.
Methods: Based on a nonsystematic literature review, the data needs of different MEA were identified. A value-based typology to classify MEA on the basis of these data needs was developed. The typology is oriented toward health outcomes and utilization and costs, key components of a new technology's value. For each MEA type, the suitability of claims data in establishing evidence of the novel technology's value in routine care was systematically assessed. Assessment criteria were data availability, completeness, timeliness, confidentiality, reliability, and validity.
Results: Claims data are better suited to MEA addressing uncertainty regarding the utilization and costs of a novel technology in routine care. In schemes where safety aspects or clinical effectiveness are assessed, the role of claims data is limited because clinical information is not included in sufficient detail.
Conclusions: The suitability of claims data depends on the source of uncertainty and, in consequence, the outcome measures chosen in the agreements. In all schemes, the validity of claims data should be judged with caution as data are collected for billing purposes. This framework may support manufacturers and payers in selecting the most suitable contract type and agreeing on contract conditions. More research is necessary to validate these results and to address remaining medical, economic, legal, and ethical questions of using claims data for MEA.
Objectives: The EuroScan International Network is a global network of publicly funded early awareness and alert (EAA) systems for health technologies. We describe the EuroScan member agency systems and methods, and highlight the potential for increased collaboration.
Methods: EuroScan members completed postal questionnaires supplemented with telephone interviews in 2012 to elicit additional information and check equivalence of responses. Information was updated between March and May 2013.
Results: Fifteen of the seventeen member agencies responded. The principal purpose of agencies is to inform decisions on coverage or reimbursement of health services and decisions on undertaking secondary research. The main users of information are national governments; health professionals; health services purchasers, commissioners, and decision makers; and healthcare providers. Most EuroScan agencies are small with almost half having fewer than two whole time equivalent staff. Ten agencies use both active and passive identification approaches, four use only active approaches. Most start identification in the experimental or investigational stages of the technology life cycle. All agencies assessed technologies when they are between the investigational and established, but under diffusion stages. Barriers to collaboration revolve around different system aims, purposes, and requirements; a lack of staff, finance, or opportunity; language differences; and restrictions on dissemination.
Conclusions: Although many barriers to collaboration were identified, the majority of agencies were supportive of increased collaboration either involving the whole EuroScan Network or between individual agencies. Despite differences in the detailed identification processes, members thought that this was the most feasible phase to develop additional collaboration.