Skip to main content Accessibility help

Disentangling Wine Judges’ Consensus, Idiosyncratic, and Random Expressions of Quality or Preference*

  • Jeffrey Bodington (a1)


Judges confer various awards on wines entered in dozens of wine competitions each year. This article employs data on blind replicates to show that those awards are based on one instance of stochastic ratings assigned by wine judges; awards based on the expected values of those stochastic ratings would be different. This article recognizes the stochastic nature of ratings and builds on the work of many others to propose and test a conditional-probability model that yields maximum-likelihood estimates of judges’ latent consensus, idiosyncratic, and random assignments of scores to wines. The exact p-value for a likelihood test of the null hypothesis that the model's results are random is less than 0.001. Applying the notion of conditional probability may lead to better methods of assigning awards to entries in wine competitions and of assessing the capabilities of wine judges. (JEL Classifications: A10, C10, C00, C12, D12)



Hide All

The author thanks an anonymous reviewer for insightful and constructive comments. All remaining errors are the responsibility of the author alone.



Hide All
Alvo, M., and Yu, P. L. H. (2014). Statistical Methods for Ranking Data. New York: Springer.
Ashton, R. H. (2012). Reliability and consensus of experienced wine judges: Expertise within and between? Journal of Wine Economics, 7(1), 7087.
Ashton, R. H. (2014). Nothing good ever came from New Jersey: Expectations and the sensory perception of wine. Journal of Wine Economics, 9(3), 304319.
Ashton, R. H. (2016). The value of expert opinion in the pricing of Bordeaux wine futures. Journal of Wine Economics, 11(2), 261288.
Bayer, P., Ferreira, F., and McMillan, R. (2003). A Unified Framework for Measuring Preferences for Schools and Neighborhoods. Yale University, Economic Growth Center, Center Discussion Paper No. 872.
Bockenholt, U. (1992). Thurstonian representation for partial ranking data. British Journal of Mathematical and Statistical Psychology, 45, 3149.
Bodington, J. C. (2015a). Evaluating wine-tasting results and randomness with a mixture of rank preference models. Journal of Wine Economics, (10)1, 3146.
Bodington, J. C. (2015b). Testing a mixture of rank preference models on judges’ scores in Paris and Princeton. Journal of Wine Economics, 10(2), 173189.
Bodington, J. C. (2017a). Wine, women, men and type II error. Journal of Wine Economics, 12(2), 161172.
Bodington, J. C. (2017b). The distribution of ratings assigned to blind replicates. Journal of Wine Economics, forthcoming.
Cao, J., and Stokes, L (2017). Comparison of different ranking methods in wine tasting. Journal of Wine Economics, 12(2), 203210.
Chen, W. (2014). How to order sushi. PhD diss., Harvard University.
Cicchetti, D. (2014). Blind Tasting of South African Wines: A Tale of Two Methodologies. American Association of Wine Economists, Working Paper No. 164.
Cicchetti, D. (2017). Evaluating the value of triplicate tastings of a given wine: Biostatistical considerations. Journal of Wine Research, 28(2), 135143.
Cleaver, G., and Wedel, M. (2001). Identifying random-scoring respondent in sensory research using finite mixture regression results. Food Quality and Preference, 12, 373384.
Cortez, P., Cerdeira, A., Almeida, F., Matos, T., and Reis, J. (2009). Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems, 47(4), 547553.
Critchlow, D. E. (1980). Metric methods for analyzing partially ranked data. Lecture notes in Statistics 34. New York: Springer-Verlag.
Frost, M. B., and Nobel, A. (2002). Preliminary study of the effect of knowledge and sensory expertise on liking for red wines. American Journal of Enology and Viticulture, 53(4), 275284. See also related UC Davis Viticulture and Enology, Summary 115.
Ginsburgh, V., and Zang, I. (2012). Shapley ranking of wines. Journal of Wine Economics, 7(2), 169180.
Greene, W. H., and Hensher, D. A. (2010). Modeling Ordered Choices. Cambridge: Cambridge University Press.
Green, P. E., and Rao, V. (1972). Applied Multidimensional Scaling: A Comparison of Approaches and Algorithms. Holt, Rienhart and Winston, Austen.
Hastings, J. S., Kane, T. J., and Staiger, D. O. (2006). Preferences and heterogeneous treatment effects in a public school choice lottery. National Bureau of Economic Research, Working Paper 12145.
Hodgson, R. T. (2008). An examination of judge reliability at a major U.S. wine competition. Journal of Wine Economics, 3(2), 105113.
Hodgson, R., and Cao, J. (2014). Criteria for accrediting expert wine judges. Journal of Wine Economics, 9(1), 6274.
Keane, M. P., and Wasi, N. (2013). The structure of consumer taste heterogeneity in revealed vs. stated preference data. University of Oxford, Nuffield College, Economics Papers from Economics Group, No. 2013-W10.
Mallows, C. L. (1957). Non-null ranking models. Biometrika, 44, 114130.
Mantonakis, A., Rodero, P., Lesschaeve, I., and Hastie, R. (2009). Order in choice: Effects of serial position on preferences. Psychological Science, 20(11), 13091312.
Marden, J. I. (1995). Analyzing and Modeling Rank Data. London: Chapman and Hall.
Mc Breen, J., Goffette-Nagot, F., and Jensen, P. (2009). An agent-based simulation of rental housing markets. Groupe d'Analyse et de Théorie Economique: Working Paper GATER 2009-08.
Nachev, A., and Hogan, M. (2013). Using data mining techniques to predict product quality from physiochemical data. Business Information Systems, Cairnes Business School, NUI, Galway, Ireland.
Olkin, I., Lou, Y., Stokes, L., and Cao, J. (2015). Analyses of wine-tasting data: A tutorial. Journal of Wine Economics, 10(1), 430.
Pearson, K. (1900). On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philosophical Magazine, 50(302), 157175.
Plackett, R. L. (1975). The analysis of permutations. Applied Statistics, 24, 193202.
Rajan, U., and Sinha, A. (2008). Equilibria in a hotelling model: First-mover advantage? Ross School of Business, Working Paper No. 1114.
Rhee, D., de Palma, A., and Thisse, J. (1998). First-mover disadvantage with consumers’ idiosyncratic preferences along unobservable characteristics. Regional Science and Economics, 36(1), 99117.
Theusen, K. F. (2007). Analysis of ranked preference data. Informatics and mathematical modeling. Master's thesis, Technical University of Denmark, Kongens Lyngby, Denmark.
Train, K. (2002). Discrete Choice Methods with Simulation. Cambridge: Cambridge University Press.
Vargo, M. D. (1989). Microbiological spoilage of a moderate acid food system using a dairy-based salad dressing model. Master's thesis, Department of Food Science and Nutrition, The Ohio State University. See also discussion in Fligner, M. A., and Verducci, J. S. (1993), Probability Models and Statistical Analyses for Ranking Data. Springer-Verlag, 1114.
Vigneau, E., Courcoux, P., and Semenou, M. (1999). Analysis of ranked preference data using latent class models. Food Quality and Preference, 10(1999), 201207.
Wine and Spirit Education Trust. (2014). Wines and spirits, looking behind the label. London: Wine and Spirit Education Trust.
Yue, C., Zhao, S., and Kuzma, J. (2015). Heterogeneous consumer preferences for nanotechnology and genetic-modification technology in food products. Journal of Agricultural Economics, 66, 308328.



Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed