Hostname: page-component-8448b6f56d-wq2xx Total loading time: 0 Render date: 2024-04-25T06:03:17.768Z Has data issue: false hasContentIssue false
  • English
  • Français

Comparison of three instruments assessing the quality of economic evaluations: A practical exercise on economic evaluations of the surgical treatment of obesity

Published online by Cambridge University Press:  04 July 2008

Sophie Gerkens ,
Ralph Crott ,
Irina Cleemput ,
Jean-Paul Thissen ,
Marie-Christine Closon ,
Yves Horsmans  and
Claire Beguin
Sophie Gerkens
Affiliation:
Université Catholique de Louvain
Ralph Crott
Affiliation:
Cliniques Universitaires Saint-Luc
Irina Cleemput
Affiliation:
Belgian Health Care Knowledge Centre (KCE)
Jean-Paul Thissen
Affiliation:
Université Catholique de Louvain and Cliniques Universitaires Saint-Luc
Marie-Christine Closon
Affiliation:
Université Catholique de Louvain
Yves Horsmans
Affiliation:
Université Catholique de Louvain and Cliniques Universitaires Saint-Luc
Claire Beguin
Affiliation:
Cliniques Universitaires Saint-Luc
Get access Rights & Permissions [Opens in a new window]

Abstract

Objectives: The increasing use of full economic evaluations has led to the development of various instruments to assess their quality. The purpose of this study was to compare the frequently used British Medical Journal (BMJ) check-list and two new instruments: the Consensus Health Economic Criteria (CHEC) list and the Quality of Health Economic Studies (QHES) instrument. The analysis was based on a practical exercise on economic evaluations of the surgical treatment of obesity.

Methods: The quality of nine selected studies was assessed independently by two health economists. To compare instruments, the Spearman rank correlation coefficient was calculated for each assessor. Moreover, the test–retest reliability for each instrument was assessed with the intraclass correlation coefficient (ICC) (3,1). Finally, the inter-rater agreement for each instrument was estimated at two levels: comparison of the total score of each article by the ICC(2,1) and comparison of results per item by kappa values.

Results: The Spearman's rank correlation coefficient between instruments was usually high (rho > 0.70). Furthermore, test–retest reliability was good for every instruments, that is, 0.98 (95 percent CI, 0.86–0.99) for the BMJ check-list, 0.97 (95 percent CI, 0.73–0.98) for the CHEC list, and 0.95 (95 percent CI, 0.75–0.99) for the QHES instrument. However, inter-rater agreement was poor (kappa < 0.40 for most items and ICC(2,1) ≤ 0.5).

Conclusions: The study shows that the results of the quality assessment of economic evaluations are not so much influenced by the instrument used but more by the assessor. Therefore, quality assessments should be performed by at least two independent experts and final scoring based on consensus.

Keywords

Quality assuranceHealth careCost and cost analysisQuestionnairesReviewSystematicTechnology assessment
Type
GENERAL ESSAYS
Information
International Journal of Technology Assessment in Health Care , Volume 24 , Issue 3 , July 2008 , pp. 318 - 325
Copyright
Copyright © Cambridge University Press 2008

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

1. Agren, G, Narbro, K, Jonsson, E, et al. Cost of in-patient care over 7 years among surgically and conventionally treated obese patients. Obes Res. 2002;10:1276–1283.CrossRefGoogle ScholarPubMed
2. Ament, A, Evers, S, Goossens, M, De Vet, H, Van Tulder, M. Criteria list for conducting systematic reviews based on economic evaluation studies – the CHEC project. In: Donaldson, C, Mugford, M, Vale, L, eds. Evidence-based health economics. From effectiveness to efficiency in systematic review. London: BMJ Books; 2002:99–113.Google Scholar
3. Au, F, Prahardhi, S, Shiell, A. Reliability of two instruments for critical assessment of economic evaluations. Value Health. 2007. In press.CrossRefGoogle Scholar
4. Chevallier, JM, Daoud, F, Szwarcensztein, K, Volcot, MF, Rupprecht, MF. Medicoeconomic evaluation of the treatment of morbid obesity by Swedish adjustable gastric banding (SAGB). Ann Chir. 2006;131:12–21.CrossRefGoogle ScholarPubMed
5. Chiou, CF, Hay, JW, Wallace, JF, et al. Development and validation of a grading system for the quality of cost-effectiveness studies. Med Care. 2003;41:32–44.CrossRefGoogle ScholarPubMed
6. Christou, NV, Sampalis, JS, Liberman, M, et al. Surgery decreases long-term mortality, morbidity, and health care use in morbidly obese patients. Ann Surg. 2004;240:416–423.CrossRefGoogle ScholarPubMed
7. Clegg, AJ, Colquitt, J, Sidhu, MK, et al. The clinical effectiveness and cost-effectiveness of surgery for people with morbid obesity: A systematic review and economic evaluation. Health Technol Assess. 2002;6:1–153.CrossRefGoogle ScholarPubMed
8. Craig, BM, Tseng, DS. Cost-effectiveness of gastric bypass for severe obesity. Am J Med. 2002;113:491–498.CrossRefGoogle ScholarPubMed
9. Drummond, MF, Jefferson, TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party. BMJ. 1996;313:275–283.CrossRefGoogle Scholar
10. Drummond, MF, Sculpher, MJ, Torrance, JW, O'Brien, BJ, Stoddart, JL. Methods for the economic evaluation of health care programmes. Oxford: Oxford University Press; 2005.CrossRefGoogle Scholar
11. Evers, S, Goossens, M, de Vet, H, van Tulder, M, Ament, A. Criteria list for assessment of methodological quality of economic evaluations: Consensus on health economic criteria. Int J Technol Assess Health Care. 2005;21:240–245.CrossRefGoogle ScholarPubMed
12. Fleiss, JL. Statistical methods for rates and proportions. New York: John Wiley and Sons; 1981.Google Scholar
13. Jefferson, T, Demicheli, V, Vale, L. Quality of systematic reviews of economic evaluations in health care. JAMA. 2002;287:2809–2812.CrossRefGoogle ScholarPubMed
14. Lambert, ML, Kohn, L, Vinck, I, et al. Pharmacological and surgical treatment of obesity. Residential care for severely obese children in Belgium. Brussels: Belgian Health Care Knowledge Centre (KCE); 2006. Report 36C.Google Scholar
15. Landis, JR, Koch, GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174.CrossRefGoogle ScholarPubMed
16. Martin, LF, Tan, TL, Horn, JR, et al. Comparison of the costs associated with medical and surgical treatment of obesity. Surgery. 1995;118:599–606.CrossRefGoogle ScholarPubMed
17. Nguyen, NT, Goldman, C, Rosenquist, CJ, et al. Laparoscopic versus open gastric bypass: A randomized study of outcomes, quality of life, and costs. Ann Surg. 2001;234:279–289.CrossRefGoogle ScholarPubMed
18. Shrout, PE, Fleiss, JL. Intraclass correlations: Uses in assessing rater reliability. Psychol Bull. 1979;86:420–428.CrossRefGoogle ScholarPubMed
19. van Gemert, WG, Adang, EM, Kop, M, et al. A prospective cost-effectiveness analysis of vertical banded gastroplasty for the treatment of morbid obesity. Obes Surg. 1999;9:484–491.CrossRefGoogle ScholarPubMed
20. van Mastrigt, GA, van Dielen, FM, Severens, JL, Voss, GB, Greve, JW. One-year cost-effectiveness of surgical treatment of morbid obesity: Vertical banded gastroplasty versus Lap-Band. Obes Surg. 2006;16:75–84.CrossRefGoogle ScholarPubMed
21. Whitman, NI. The Delphi technique as an alternative for committee meetings. J Nurs Educ. 1990;29:377–379.CrossRefGoogle ScholarPubMed
Supplementary material: File

Gerkens supplementary material

Gerkens supplementary material

Download Gerkens supplementary material(File)
File 106.5 KB