Skip to main content Accessibility help

An Analysis of Wine Critic Consensus: A Study of Washington and California Wines*

  • Eric T. Stuen (a1), Jon R. Miller (a2) and Robert W. Stone (a3)


We examine the degree of consensus in quality ratings of prominent U.S. wine publications. For the purposes of wine consumption and research, are ratings on the ubiquitous 100-point scale reliable measures of quality? The value of expert judgment has been called into question by a number of studies, especially in the context of wine competitions and tasting events. Using data on 853 wines, we find a moderately high level of consensus, measured by the correlation coefficient, between most pairs of publications, similar to the level found by Ashton (2013). Rank and intraclass correlations are similar. Consensus is not found to be related to the blinding policies (or lack thereof) of the critical publications. (JEL Classifications: C93, D46)


Corresponding author

e-mail: (corresponding author).


Hide All

We thank an anonymous referee for valuable comments. We also thank Dominic Cicchetti, Omer Gokcekus, Robin Goldstein, Robert Hodgson, Lynne Stokes, and other participants at the 2014 AAWE Annual Conference in Walla Walla, WA, for helpful comments and Stefan Richardson and Matthew Hill for excellent research assistance.



Hide All
Ali, H.H., Lecocq, S., and Visser, M. (2008). The impact of gurus: Parker grades and en primeur wine prices. Economic Journal, 118(529), F158F173.
Ashenfelter, O. (2006). Tales from the crypt: Bruce Kaiser tells about the trials and tribulations of a wine judge. Journal of Wine Economics, 1(2), 173175.
Ashenfelter, O., and Quandt, R. (1999). Analyzing a wine tasting statistically. Chance, 12(3), 1620.
Ashenfelter, O., and Storchmann, K. (2012). Editorial: The Judgment of Princeton and other articles. Journal of Wine Economics, 7(2), 139142.
Ashton, R.H. (2011). Improving experts’ wine quality judgments: Two heads are better than one. Journal of Wine Economics, 6(2), 160178.
Ashton, R.H. (2012). Reliability and consensus of experienced wine judges: Expertise within and between? Journal of Wine Economics, 7(1), 7087.
Ashton, R.H. (2013). Is there consensus among wine quality ratings of prominent critics? An empirical analysis of red Bordeaux, 2004–2010. Journal of Wine Economics, 8(2), 225234.
Ashton, R.H. (2014). Wine as an experience good: Price versus enjoyment in blind tastings of expensive and inexpensive wines. Journal of Wine Economics, 9(2), 171182.
Bodington, J. (2012). 804 tastes: Evidence on preferences, randomness, and value from double-blind wine tastings. Journal of Wine Economics, 7(2), 181191.
Brien, C.J., May, P., and Mayo, O. (1987). Analysis of judge performance in wine-quality evaluations. Journal of Food Science, 52(5), 12731279.
Cao, J., and Stokes, L. (2010). Evaluation of wine judge performance through three characteristics: Bias, discrimination, and variation. Journal of Wine Economics, 5(1), 132142.
Cicchetti, D.V. (2004a). Who won the 1976 blind tasting of French Bordeaux and U.S. cabernets? Parametrics to the rescue. Journal of Wine Research, 15(3), 211220.
Cicchetti, D.V. (2004b). On designing experiments and analyzing data to assess the reliability and accuracy of blind tastings. Journal of Wine Research, 15(3), 221226.
Cicchetti, D. (2006). The Paris 1976 wine tastings revisited once more: Comparison of ratings of consistent and inconsistent tasters. Journal of Wine Economics, 1(2), 125140.
Cliff, M.A., and King, M.C. (1997). Application of eggshell plots for evaluation of judges at wine competitions. Journal of Wine Research, 8, 7580.
Gawel, R., and Godden, P.W. (2008). Evaluation of the consistency of wine quality assessment from expert wine tasters. Australian Journal of Grape and Wine Research, 14(1), 18.
Ginsburgh, V., and Zang, I. (2012). Shapley ranking of wines. Journal of Wine Economics, 7(2), 169180.
Gokcekus, O., and Nottebaum, D. (2011). The buyer's dilemma—Whose rating should a wine drinker pay attention to? AAWE Working Paper 91.
Goldstein, R., Almenberg, J., Dreber, A., Emerson, J.W., Herschkowitsch, A., and Katz, J. (2008). Do more expensive wines taste better? Evidence from a large sample of blind tastings. Journal of Wine Economics, 3(1), 19.
Hodgson, R.T. (2008). An examination of judge reliability at a major U.S. wine competition. Journal of Wine Economics, 3(2), 105113.
Hodgson, R.T. (2009a). An analysis of the concordance among 13 U.S. wine competitions. Journal of Wine Economics, 4(1), 19.
Hodgson, R.T. (2009b). How expert are “expert” wine judges? Journal of Wine Economics, 4(2), 233241.
Lawless, H., Liu, Y., and Goldwyn, C. (1997). Evaluation of wine quality using a small-panel hedonic scaling method. Journal of Sensory Studies, 12(4), 317332.
McGraw, K.O., and Wong, S.P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 3046.
Miller, J., Genc, I., and Driscoll, A. (2007). Wine price and quality: in search of a signaling equilibrium in 2001 California cabernet sauvignon. Journal of Wine Research, 18(1), 3546.
Olkin, I., Lou, Y., Stokes, L., and Cao, J. (2014). Analyses of Wine-Tasting Data: A Tutorial. Journal of Wine Economics, 10(1), doi:10.1017/jwe.2014.26.
Quandt, R.E. (2012). Comments on the Judgment of Princeton. Journal of Wine Economics, 7(2), 152154.
Taber, G.M. (2012). The Judgment of Princeton. Journal of Wine Economics, 7(2), 143151.
Ward, D.L. (2012). A graphical and statistical analysis of the Judgment of Princeton wine tasting. Journal of Wine Economics, 7(2), 155168.
Weil, R.L. (2005). Analysis of reserve and regular bottlings: Why pay for a difference no critics claim to notice? Chance, 18(3), 915.



Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed