Skip to main content Accessibility help

Assessment of observer performance in a subjective scoring system: visual classification of the gait of cows

  • B. ENGEL (a1), G. BRUIN (a1), G. ANDRE (a2) and W. BUIST (a1)


As with any measurement procedure, the performance of a subjective classification procedure must be evaluated. Observers have to be trained and their performance has to be assessed, preferably on a regular basis, to guarantee sufficient consistency and accuracy of classification results. The current paper is a study of observer performance where observers were asked to classify the gait of cows from video recordings. Gait was classified in nine ordered categories (ranging from 1=normal gait to 9=severely abnormal gait) and also as a continuous fraction by putting a mark on a paper strip (the left end corresponding to 0=normal gait and the right end to 1=severely abnormal gait). The use of statistical models and methodology for analysis of these visual scores is demonstrated and discussed. Observers were assessed by comparing their classification results with the results of an expert. Models and methodology take proper account of typical features of the data, i.e. the fact that data are discrete scores or continuous scores with an upper and lower bound, the variance heterogeneity and non-linearity of model terms that arises from this, and the dependence between repeated classifications of videos of the same cow. Results of the analyses are summarized in simple tables and plots. These are useful tools to indicate possible flaws in judgement of an observer, that may be corrected by further training. When a high standard is developed, which usually takes the form of the opinion of one or more experts, this methodology can be applied prior to any experiment where responses are ordered subjective scores.


Corresponding author

Biometris, P.O. Box 100, 6700 AC Wageningen, The Netherlands. Email:
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

The Journal of Agricultural Science
  • ISSN: 0021-8596
  • EISSN: 1469-5146
  • URL: /core/journals/journal-of-agricultural-science
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed