Skip to main content Accessibility help

Coverage with evidence development: The Ontario experience

  • Leslie Levin (a1), Ron Goeree (a2), Mark Levine (a3), Murray Krahn (a4), Tony Easty (a5), Adalstein Brown (a6) and David Henry (a7)...


Background: For non-drug technologies, there is often residual uncertainty following systematic review, mainly due to inadequate evidence of efficacy. The unwillingness to make decisions in the presence of uncertainty may lead to passive diffusion and intuitive decision making with or without public pressure. This may affect health system sustainability. There is increasing interest in post-market evaluation through processes that include coverage with evidence development (CED) to address residual uncertainty regarding effectiveness and cost-effectiveness. Global experience of CED has been slow to develop despite their potential contribution to decision making.

Methods: Ontario's field evaluation program to better inform decision making represents a collaboration between physicians, policy decision makers and academic centers. We report results of the first ten CEDs from this program to assess whether they achieved their objective of influencing policy by addressing residual uncertainty following systematic review.

Results: Since 2003, nineteen field evaluation studies to resolve residual uncertainty following systematic review have been completed, ten of which met the criteria of CED and are the focus of this report. There was more than one patient subgroup or intervention in three of the CEDs. This provided the basis for evaluating thirteen outcomes. In each case, the CED addressed the uncertainty and led to a decision based on the systematic review and CED result. The CEDs led to adoption of the technology in six instances, modified adoption in three instances and withdrawal in four instances.

Conclusions: CED makes an important contribution to translating evidence to decision making. Methodologies are needed to increase the scope and reduce timelines for CEDs, such as the use of linked comprehensive and robust data sets and collaborative studies with other jurisdictions. CED before making long-term funding decisions, especially where there is uncertainty of effectiveness, safety or cost-effectiveness, should be increasingly funded by health systems.



Hide All
1. Atkins, D, Best, D, Briss, PA, et al. Grading quality of evidence and strength of recommendations. BMJ. 2004;328:1490.
2. Chalkidou, K, Hoy, A, Littlejohns, P. Making a decision to wait for more evidence: When the National Institute for Health and Clinical Excellence recommends a technology only in the context of research. J R Soc Med. 2007;100:453460.
3. Chalkidou, K, Tunis, S, Lopert, R, et al. Comparative effectiveness research and evidence-based health policy: Experience from four countries. Milbank Q. 2009;87:339367.
4. Chow, BJW, Freeman, MR, Bowen, JM, et al. For the OMCAS Investigators. Ontario Multi-Director Computed Tomographic Coronary Angiography Study (OMCAS): A field evaluation of diagnostic accuracy. Arch Intern Med. (in press).
5. Clarke, PM, Gray, AM, Briggs, A, et al. A model to estimate the lifetime health outcomes of patients with type 2 diabetes: The United Kingdom Prospective Diabetes Study (UKPDS) Outcomes Model (UKPDS no. 68). Diabetologia. 2004;47:17471759.
6. Claxton, K, Briggs, A, Buxton, MJ, et al. Value based pricing for NHS drugs: An opportunity not to be missed? BMJ. 2008;336:251254.
7. Evans, WK, Laupacis, A, Gulenchyn, KY, Levin, L, Levine, M. Evidence-based approach to the introduction of positron emission tomography in Ontario, Canada. J Clin Oncol. 2009;27:56075613.
8. Feder, BJ. Doctors rethink widespread use of heart stents. October 21, 2006. The New York Times. (accessed December 2, 2009).
9. Goeree, R, Levin, L. Building bridges between academic research and policy formulation: The PRUFE framework – an integral part of Ontario's evidence-based HTPA process. Pharmacoeconomics. 2006;24:11431156.
10. Institute for Clinical Evaluative Sciences. Health technology assessment of positron emission tomography (PET) – A systematic review. May 2001. (accessed December 2, 2009).
11. Johnson, AP, Sikich, NJ, Evans, G, et al. Health technology assessment: A comprehensive framework for evidence-based recommendations in Ontario. Int J Technol Assess Health Care. 2009;25:141150.
12. Levin, L, Goeree, R, Sikich, N, et al. Establishing a comprehensive continuum from an evidentiary base to policy development for health technologies: The Ontario experience. Int J Technol Assess Health Care. 2007;23:299309.
13. Maziak, DE, Darling, GE, Inculet, RI, et al. Positron emission tomography in staging early lung cancer: A randomized trial. Ann Intern Med. 2009;151:221248.
14. Medical Advisory Secretariat. Endovascular repair of abdominal aortic aneurysm: An evidence-based analysis. Ontario Health Technol Assess Ser. [Internet]. 2002;2:1–46.
15. Medical Advisory Secretariat. Non-invasive cardiac imaging technologies for the diagnosis of coronary artery disease: A summary of evidence-based analyses. Ont Health Technol Assess Ser. [Internet]. 2010;10:1–40.
16. Medical Advisory Secretariat. Diabetes strategy evidence platform: A summary of evidence-based analyses. Ont Health Technol Assess Ser. 2009;9:143.
17. Morice, MC, Serruys, PW, Sousa, JE, et al. A randomized comparison of a sirolimus-eluting stent with a standard stent for coronary revascularization. N Engl J Med. 2002;346:17731780.
18. Moses, JW, Leon, MB, Popma, JJ, et al. Sirolimus-eluting stents versus standard stents in patients with stenosis in a native coronary artery. N Engl J Med. 2003;349:13151323.
19. National Institute for Health and Clinical Excellence. Research and development strategy. London: National Institute for Health and Clinical Excellence; 2004. (accessed Decem-ber 2, 2009).
20. Neumann, PJ, Tunis, SR. Medicare and medical technology—the growing demand for relevant outcomes. N Engl J Med. 2010;362:377379.
21. O'Reilly, D, Hopkins, R, Blackhouse, G, et al. Long-term cost-utility analysis of a multidisciplinary primary care diabetes management program in Ontario. Can J Diabetes. 2007;31:205214.
22. Pritchard, KI, Julian, J, McCready, D. A prospective study evaluating 18F-Fluorodeoxyglucose (18FDG) positron emission tomography (PET) in the assessment of axillary nodal spread in women undergoing sentinel lymph node biopsy (SLNB) for breast cancer. J Clin Oncol. 2008;26 (Suppl):533.
23. Stadtmauer, EA, O'Neill, A, Goldstein, LJ, et al. Conventional-dose chemotherapy compared with high-dose chemotherapy plus autologous hematopoietic stem-cell transplantation for metastatic breast cancer. Philadelphia Bone Marrow Transplant Group. N Engl J Med. 2000;342:10691076.
24. Tanenbaum, SJ. “Medical effectiveness” in Canadian and U.S. health policy: The comparative politics of inferential ambiguity. Health Serv Res. 1996;31:517532.
25. Tarride, JE, Blackhouse, G, De Rose, G, et al. Cost-effectiveness analysis of elective endovascular repair compared with open surgical repair of abdominal aortic aneurysms for patients at a high surgical risk: A 1-year patient-level analysis conducted in Ontario, Canada. J Vasc Surg. 2008;48:779787.
26. Tu, JV, Bowen, J, Chiu, M, et al. Effectiveness and safety of drug-eluting stents in Ontario. N Engl J Med. 2007;357:13931402.
27. Tunis, SR, Pearson, SD. Coverage options for promising technologies: Medicare's ‘coverage with evidence development’. Health Aff (Millwood). 2006;25:12181230.
28. Tunis, SR, Stryer, DB, Clancy, CM. Practical clinical trials: Increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003;290:16241632.
29. Ung, Y, Sun, A, MacRae, R. Impact of positron emission tomography (PET) in stage III non-small cell lung cancer (NSCLC): A prospective randomized trial (PET START). J Clin Oncol. 2009;27: 15(Suppl):7548.
30. Whicher, DM, Chalkidou, K, Dhalla, IA, Levin, L, Tunis, S. Comparative effectiveness research in Ontario, Canada: Producing relevant and timely information for health care decision makers. Milbank Q. 2009;87:585606.


Related content

Powered by UNSILO

Coverage with evidence development: The Ontario experience

  • Leslie Levin (a1), Ron Goeree (a2), Mark Levine (a3), Murray Krahn (a4), Tony Easty (a5), Adalstein Brown (a6) and David Henry (a7)...


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed.