Published online by Cambridge University Press: 08 June 2012
The objective of this research is to devise a reliable and accurate scoring algorithm for awarding medals at a major U.S. wine competition that was described earlier by Hodgson (2008). The new algorithm is comprised of a two-tiered method that considers an award on the basis of a total medal score averaged across four judges per each evaluated wine; and a reliability algorithm that grades the degree of inter-judge reliability according to the following criteria: <70% = Poor or Unacceptable; 70%–79% = Fair; 80%–89% = Good; and 90%–100% = Excellent agreement (Cicchetti, 2001). The method has advantages over a previous scoring method that based its decision upon incorrect estimates of majority opinion; and unacceptable levels of inter-judge agreement. (JEL Classification: C1, L15, Q13)