Skip to main content Accessibility help

Comparison of three instruments assessing the quality of economic evaluations: A practical exercise on economic evaluations of the surgical treatment of obesity

  • Sophie Gerkens (a1), Ralph Crott (a2), Irina Cleemput (a3), Jean-Paul Thissen (a4), Marie-Christine Closon (a1), Yves Horsmans (a4) and Claire Beguin (a2)...


Objectives: The increasing use of full economic evaluations has led to the development of various instruments to assess their quality. The purpose of this study was to compare the frequently used British Medical Journal (BMJ) check-list and two new instruments: the Consensus Health Economic Criteria (CHEC) list and the Quality of Health Economic Studies (QHES) instrument. The analysis was based on a practical exercise on economic evaluations of the surgical treatment of obesity.

Methods: The quality of nine selected studies was assessed independently by two health economists. To compare instruments, the Spearman rank correlation coefficient was calculated for each assessor. Moreover, the test–retest reliability for each instrument was assessed with the intraclass correlation coefficient (ICC) (3,1). Finally, the inter-rater agreement for each instrument was estimated at two levels: comparison of the total score of each article by the ICC(2,1) and comparison of results per item by kappa values.

Results: The Spearman's rank correlation coefficient between instruments was usually high (rho > 0.70). Furthermore, test–retest reliability was good for every instruments, that is, 0.98 (95 percent CI, 0.86–0.99) for the BMJ check-list, 0.97 (95 percent CI, 0.73–0.98) for the CHEC list, and 0.95 (95 percent CI, 0.75–0.99) for the QHES instrument. However, inter-rater agreement was poor (kappa < 0.40 for most items and ICC(2,1) ≤ 0.5).

Conclusions: The study shows that the results of the quality assessment of economic evaluations are not so much influenced by the instrument used but more by the assessor. Therefore, quality assessments should be performed by at least two independent experts and final scoring based on consensus.



Hide All
1. Agren, G, Narbro, K, Jonsson, E, et al. Cost of in-patient care over 7 years among surgically and conventionally treated obese patients. Obes Res. 2002;10:12761283.
2. Ament, A, Evers, S, Goossens, M, De Vet, H, Van Tulder, M. Criteria list for conducting systematic reviews based on economic evaluation studies – the CHEC project. In: Donaldson, C, Mugford, M, Vale, L, eds. Evidence-based health economics. From effectiveness to efficiency in systematic review. London: BMJ Books; 2002:99113.
3. Au, F, Prahardhi, S, Shiell, A. Reliability of two instruments for critical assessment of economic evaluations. Value Health. 2007. In press.
4. Chevallier, JM, Daoud, F, Szwarcensztein, K, Volcot, MF, Rupprecht, MF. Medicoeconomic evaluation of the treatment of morbid obesity by Swedish adjustable gastric banding (SAGB). Ann Chir. 2006;131:1221.
5. Chiou, CF, Hay, JW, Wallace, JF, et al. Development and validation of a grading system for the quality of cost-effectiveness studies. Med Care. 2003;41:3244.
6. Christou, NV, Sampalis, JS, Liberman, M, et al. Surgery decreases long-term mortality, morbidity, and health care use in morbidly obese patients. Ann Surg. 2004;240:416423.
7. Clegg, AJ, Colquitt, J, Sidhu, MK, et al. The clinical effectiveness and cost-effectiveness of surgery for people with morbid obesity: A systematic review and economic evaluation. Health Technol Assess. 2002;6:1153.
8. Craig, BM, Tseng, DS. Cost-effectiveness of gastric bypass for severe obesity. Am J Med. 2002;113:491498.
9. Drummond, MF, Jefferson, TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party. BMJ. 1996;313:275283.
10. Drummond, MF, Sculpher, MJ, Torrance, JW, O'Brien, BJ, Stoddart, JL. Methods for the economic evaluation of health care programmes. Oxford: Oxford University Press; 2005.
11. Evers, S, Goossens, M, de Vet, H, van Tulder, M, Ament, A. Criteria list for assessment of methodological quality of economic evaluations: Consensus on health economic criteria. Int J Technol Assess Health Care. 2005;21:240245.
12. Fleiss, JL. Statistical methods for rates and proportions. New York: John Wiley and Sons; 1981.
13. Jefferson, T, Demicheli, V, Vale, L. Quality of systematic reviews of economic evaluations in health care. JAMA. 2002;287:28092812.
14. Lambert, ML, Kohn, L, Vinck, I, et al. Pharmacological and surgical treatment of obesity. Residential care for severely obese children in Belgium. Brussels: Belgian Health Care Knowledge Centre (KCE); 2006. Report 36C.
15. Landis, JR, Koch, GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159174.
16. Martin, LF, Tan, TL, Horn, JR, et al. Comparison of the costs associated with medical and surgical treatment of obesity. Surgery. 1995;118:599606.
17. Nguyen, NT, Goldman, C, Rosenquist, CJ, et al. Laparoscopic versus open gastric bypass: A randomized study of outcomes, quality of life, and costs. Ann Surg. 2001;234:279289.
18. Shrout, PE, Fleiss, JL. Intraclass correlations: Uses in assessing rater reliability. Psychol Bull. 1979;86:420428.
19. van Gemert, WG, Adang, EM, Kop, M, et al. A prospective cost-effectiveness analysis of vertical banded gastroplasty for the treatment of morbid obesity. Obes Surg. 1999;9:484491.
20. van Mastrigt, GA, van Dielen, FM, Severens, JL, Voss, GB, Greve, JW. One-year cost-effectiveness of surgical treatment of morbid obesity: Vertical banded gastroplasty versus Lap-Band. Obes Surg. 2006;16:7584.
21. Whitman, NI. The Delphi technique as an alternative for committee meetings. J Nurs Educ. 1990;29:377379.


Type Description Title
Supplementary materials

Gerkens supplementary material
Gerkens supplementary material

 Word (106 KB)
106 KB

Comparison of three instruments assessing the quality of economic evaluations: A practical exercise on economic evaluations of the surgical treatment of obesity

  • Sophie Gerkens (a1), Ralph Crott (a2), Irina Cleemput (a3), Jean-Paul Thissen (a4), Marie-Christine Closon (a1), Yves Horsmans (a4) and Claire Beguin (a2)...


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed