Skip to main content Accessibility help
×
Home

An assessment of inter-rater agreement of the literature filtering process in the development of evidence-based dietary guidelines

  • Marcia Cooper (a1) (a2), Wendy Ungar (a3) and Stanley Zlotkin (a1) (a4) (a5)

Abstract

Objective

To determine whether the literature filtering process, a vital initial component of a systematic literature review, could be successfully completed by nutrition professionals or non-professionals.

Design

Using a diet–disease relationship as the guideline topic, inter-rater agreement for the title and abstract filtering processes between and among professionals and non-professionals was assessed and compared with an expert reference standard. Predetermined eligibility criteria were applied by all raters to 185 titles and 90 abstracts. Filtering decisions were initially made independently and then revised after a within-pair consensus meeting.

Subjects

The raters were six dietitians (RD) and six nutrition graduate students (Grad). To assess inter-rater agreement (reliability), each group was divided into three pairs.

Results

Weighted and unweighted kappa statistics and percentage agreement were calculated to determine the inter-rater agreement within pairs. Sensitivity and specificity estimates were determined by comparing responses with those of an expert reference standard. Overall, Grad pairs demonstrated greater inter-rater agreement than RD pairs for title filtering (P < 0.05); no differences were observed for abstract filtering. Compared with the expert reference standard, every rater and pair had false-negative responses for both title and abstract filtering.

Conclusions

After consensus meetings, both RDs and Grads were comparable in their agreement on title and abstract filtering, although important differences remained compared with the expert reference standard. This study provides preliminary findings on the value of utilising a non-expert pair in developing guidelines, and suggests that the literature filtering process is complex and quite subjective.

    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      An assessment of inter-rater agreement of the literature filtering process in the development of evidence-based dietary guidelines
      Available formats
      ×

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      An assessment of inter-rater agreement of the literature filtering process in the development of evidence-based dietary guidelines
      Available formats
      ×

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      An assessment of inter-rater agreement of the literature filtering process in the development of evidence-based dietary guidelines
      Available formats
      ×

Copyright

Corresponding author

*Corresponding author: Email marcia_cooper@hc-sc.gc.ca

References

Hide All
1Cochrane Collaboration Handbook. In: Mulrow, CD, Oxman, AD, eds. The Cochrane Library, 4th ed [database on disk and CD-ROM]. Oxford: The Cochrane Collaboration, Oxford Update Software, 1997.
2Splett, P. Developing and Validating Evidence-Based Guides for Practice: A Tool Kit for Dietetics Professionals. Chicago, IL: American Dietetic Association, 1999.
3Brunner, E, Rayner, M, Thorogood, M, Margetts, B, Hooper, L, Summerbell, C, et al. Making public health nutrition relevant to evidence-based action. Public Health Nutrition 2001; 4: 1297–9.
4Streiner, DL. Learning how to differ: agreement and reliability statistics in psychiatry. Canadian Journal of Psychiatry 1995; 40: 60–6.
5Haas, M. Statistical methodology for reliability studies. Journal of Manipulative and Physiological Therapeutics 1991; 14: 1932.
6Cohen, J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960; 20: 3746.
7Fitzmaurice, G. Statistical methods for assessing agreement. Nutrition 2002; 18: 694–6.
8Rigby, AS. Statistical methods in epidemiology. V. Towards an understanding of the kappa coefficient. Disability and Rehabilitation 2000; 22: 339–44.
9Cicchetti, DV, Allison, T. A new procedure for assessing reliability of scoring EEG sleep recordings. American Journal of EEG Technology 1971; 11: 101–9.
10Landis, JR, Koch, JJ. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159–74.
11Fletcher, RH, Fletcher, SW, Wagner, EH. Clinical Epidemiology: The Essentials, 2nd ed. London: Williams and Wilkins, 1988.
12Gordis, L. Epidemiology. Philadelphia, PA: WB Saunders Co., 2000.
13Cooper, MJ, Zlotkin, SH. An evidence-based approach to the development of national dietary guidelines. Journal of the American Dietetic Association 2003; 103: S2833.
14Margetts, BM, Vorster, HH, Venter, CS. Evidence-based nutrition. South African Journal of Clinical Nutrition 2002; 15: 712.
15Oxman, AD, Guyatt, GH, Cook, DJ, Jaeschke, R, Heddle, N, Keller, J. An index of scientific quality for health reports in the lay press. Journal of Clinical Epidemiology 1993; 46: 9871001.
16Arrive, L, Renard, R, Carrat, F, Belkacem, A, Dahan, H, Le Hir, P. A scale of methodological quality for clinical studies of radiologic examinations. Radiology 2000; 217: 6974.
17Moher, D, Cook, DJ, Eastwood, S, Olkin, I, Rennie, D, Stroup, DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 1999; 354: 1896–900.
18Moher, D, Schulz, KF, Altman, DG; CONSORT group. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. Journal of the American Podiatric Medical Association 2001; 91: 437–42.
19Sands, ML, Murphy, JR. Use of kappa statistic in determining validity of quality filtering for meta-analysis: a case study of the health effects of electromagnetic radiation. Journal of Clinical Epidemiology 1996; 49: 1045–51.

Keywords

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed