who judges the judges?

Tuesday, September 01, 2009

Dr. Karl Storchmann, who teaches economics at Whitman College, is also the founder of the Journal of Wine Economics. It’s a technical, often impenetrably geeky academic journal, but occasionally one of the contributors hits on a juicy topic of general interest, such as last summer when it was revealed that the Wine Spectator had given out an award to a restaurant that didn’t exist.

The latest issue of the Journal (Vol. 4, No. 1) will be released tomorrow, and has another promising lead story.


In it, author Robert T. Hodgson analyzes the reliability of gold medals awarded at 13 California Wine Fairs. “An analysis of over 4000 wines entered in 13 U.S. wine competitions shows little concordance among the venues in awarding gold medals. Of the 2,440 wines entered in more than three competitions, 47 percent received gold medals, but 84 percent of these same wines also received no award in another competition. Thus, many wines that are viewed as extraordinarily good at some competitions are viewed as below average at others. An analysis of the number of gold medals received in multiple competitions indicates that the probability of winning a gold medal at one competition is stochastically independent of the probability of receiving a gold at another competition, indicating that winning a gold medal is greatly influenced by chance alone.”

In other words, one good gold does not necessarily deserve another.

Now, I have not yet waded through the entire article, but even this brief extract raises some interesting questions. I wonder how many of the wines were entered in multiple competitions? If a wine isn’t entered, it’s not going to win anything. But more importantly, I don’t think that the buying public – you consumers – realize that these judgings, which number in the hundreds, are a long way from the impartial reviews that they pretend to be. First of all, the wineries pay hefty fees to enter them. A lot of really good wines are never entered – do you think Leonetti and Quilceda Creek need or care about gold medals at this point?

Those that are entered have a diluted playing field against which to compete. Often, the organizers, cognizant of the large fees that they are raking in, have quotas for various medals. Judges are given strict marching orders: we would like to see X percent of golds, X percent of silvers, etc.

Judges are a mixed lot at these events, and there are more than a few people who simply like the free food, free wine, free travel that goes along with being invited to judge. In the intense days of the tastings, in which well over 100 wines (often double that) are tasted by a single panel, there is precious little time to really understand any given wine. Worse yet, a judge must quickly calibrate with other judges serving together on a panel, and rare indeed is the panel (and I’ve been on many) that contained a full slate of competent palates.

So when you walk into a tasting room adorned with medals, beware. They are money-makers for the organizers of these events, and marketing tools for the wineries. But it’s a real stretch to say that they mean anything at all otherwise, unless a specific wine has received multiple awards from several of the best competitions. And that, at least according to this article, is not likely to happen.

The full article can be accessed free of charge at http://tinyurl.com/ltl767


JW said...

I think it is important to understand that while winning one gold medal may be a "fluke," consistently winning medals over a significant period of time, even if we adhere to the probabilities suggested in this article, most likely indicates quality.

Many of the wineries in Washington State have been consistently winning awards for a long time now. Barnard Griffin, Kiona, Cougar Crest, and Kestrel all come to mind.

Andy Plymale said...

There must be a music analogy, such as listening to short clips of a hundred recordings in one sitting, vs. listening to a recording in it's entirety, and in the context of a larger recording project. Just one idea.

Wawineman said...

I don't bother with awards when it comes to buying wines. I think they are all grossly biased via (1) rigged, either through sponsorship/huge "entrance fees"; (2) judges who are not of "fair palate" or of local celebrity, which in turn helps in advertising the event; (3) limited sample of wineries; and/or (4) no transparency in how the decisions were made. Oh yes, and what of this "double gold" garbage??

I look for a "judge" who consistently (1) samples the territorial realm, (2) displays transparency in wine knowledge, and (3) is accessible to input from the public. That's why PG's "Top 100 wines" at years' end is my primary guidance in accumulating the good stuff for my cellar.
The next step is a judging panel of actual winemakers with a Master recognition (Wine/Sommelier) and with a better rating system.

Post a Comment

Your comment is awaiting moderation and will be posted ASAP. Thanks!