News feeds
E-mail alerts
Mobile
Text alerts
Twitter
Facebook
Home > Blogs > Uncorked > Archives > 2006 > November > 01 > Entry
Should you trust Consumer Reports wine ratings?
The December issue of Consumer Reports magazine wades into the wine-ratings war with some subtle barbs directed at other wine consumer publications and high praise indeed for its own methodology.
But let’s take a closer look.
(As this is written, there is no link to the wine story available yet on the magazine’s “current issue” portion of its web site, which still lists November’s features and mentions the wine report is “coming soon.”)
Consumer Reports President Jim Guest writes in his opening commentary entitled “Judging wine with another scale” that the magazine chooses wines to evaluate that did well in previous tests, are produced in large-enough quantities to be widely available, and are reasonably priced. And it also includes wines that “have generated a buzz in other publications.”
Hmm. Interesting selection criteria.
Guest goes on to say that the magazine chooses not to use a 100-point scale to rate wines, but instead uses a rating system from “poor” to “excellent.” Then he lobs this little grenade:
You won’t, however, read “Consumer Reports rates this wine ‘excellent’” on (any wine store) shelves or in any advertising, since we don’t allow the use of our Ratings to promote products. Nor do we run ads in our magazine, so you’ll never see a glossy wine ad opposite our Ratings of the same wine.”
Is he suggesting that scores in other publications are bought and paid for? Hmmm.
And don’t you love the way the magazine capitalizes the term “Ratings” when it refers to its own? Almost gives CR’s evaluations a certain, shall we say, Biblical authority, don’t you think?
CR’s tasting methodology appears rigorous: “Secret shoppers” buy the wines anonymously at retail, rather than accepting free bottles from the winery, “so there’s no chance that we’ll get a bottle specially selected to impress us.” The wines are tasted blind, and every wine is tasted four times, each from a different bottle. This does sound more thorough than methods used by other wine-rating publications.
But who, might you ask, is really doing the tasting?
That remains a bit of a mystery.
“We enlist the trained palates of two wine experts with almost 60 years of combined experience,” Guest says, without identifying who these experts are.
I don’t know about you, but I can think of a several people with decades of experience tasting wines whose opinions I wouldn’t put two cents worth of stock into. But I guess we’re supposed to trust this pair of mystery tasters.
Guest says in explaining why the magazine chose not to use the 100-point scale that the magazine, as well as the two tasting experts, “believe there’s too much subjectivity in wine tasting to declare a bottle a 90, say, instead of an 89.”
But there’s not too much subjectivity to two — count ‘em, TWO — palates determining whether a wine is “very good” or “excellent?” Heck, when the magazine rates restaurant chains and grocery store chains, it uses the opinions of thousands of its subscribers nationwide who fill out a CR survey. But its wine ratings depend on exactly two palates belonging to folks we know nothing about?
Hmmm….
For the record, the magazine awarded “excellent” ratings to three chardonnays — 2004 Edna Valley Paragon San Luis Obispo County, ‘04 Kendall-Jackson Grand Reserve Monterey-Santa Barbara Counties, and ‘04 Beringer Napa Valley, all priced from $14 to $20 in the magazine — and to two zinfandels, the ‘04 Seghesio Family Vineyards Sonoma County ($20) and the ‘04 Cellar No. 8 ($10, a “CR Best Buy). No cabernet sauvignons received a score of “excellent,” though the ‘02 Columbia Crest Grand Estates Columbia Valley collected a “CR Best Buy” designation for its “very good” rating and its $11 price tag.
What do you think? How much faith do you put in CR’s ratings? Would its evaluations affect your buying decisions any more or any less than Parker’s or Wine Spectator’s?
Thanks and cheers!
Mark Fisher
Comments
By Jon Fox
November 10, 2006 4:48 PM | Link to this
I have been reading and following Consumer Reports for many years. I know a lot about consumer products and which brands and models are best, and like to see how Consumer Reports compares, and they are usually way off. Most of the time, the “best buy” or best brand or model overall, will not even be INCLUDED anywhere in the list at all! Because if you can’t buy it at Walmart or Best Buy, it doesn’t get in the ratings. More important though, is that when it comes to a TV set or stereo, one unit CAN be better than another by actual specs and fact, whereas food and wine can not be rated this way, they are only OPINIONS and PERSONAL TASTE. When it comes to what wines are best, or what movies are best, I find that reviewers never tend to agree with my personal tastes. Consumer Reports has so far stayed away from reviewing what movies are best, and perhaps they should stay out of wine too. Interesting they only rated like 3 types of wine, assuming everyone only drinks those types. What about Rieslings and a few others than the 3 they picked?By Jen
November 7, 2006 9:32 PM | Link to this
What do I think of Consumer Report’s wine ratings. Well, let’s see. (1) Blind taste tests by trained testers - GOOD. The names of the tasters is irrelevant. I’m sure the trained wine testers are quite competant to rate the wines. (2) Wines purchased by Consumer Reports - GOOD and (3) No advertising policy - GOOD. Ensures subtle influence of gifts and advertisers. (4) The ratings themselves - GOOD. I have tried the wines rated “Best Buys” and I like them all. The wines are accurately evaluated, in my opinion. (5) Rating wines that are widely available - GOOD. The ratings would be useless if I can’t buy the highly-rated wines on a regular basis. (6) Ratings presented in a useable format - GOOD. Like the famous Siskle & Ebert “Thumbs Up” or “Thumbs down”, the CR Ratings are extremely usable. I agree that there is no statistically significant difference between an 89 and a 90 on a 100-point scale. Likewise, taking economics into account helps me to find the best buys quickly. In conclusion, I think the ratings are wonderful and have already been very useful. I alerted my local grocery store’s liquor purchaser to be sure to have all these wines on hand.By Rob
November 4, 2006 1:25 PM | Link to this
CR is focused on the general public,where as the other wine oriented journals are focused on individuals who take their wine more seriously. As a guide for people who would like to get a reasonably good wine, without investing a lot of time and money, CR is perfect for them. This will often lead them to better wines, with some confidence, of what an acceptable wine should be like.By Jack at F&B
November 4, 2006 10:48 AM | Link to this
Mark Fisher is right; you need to know who the two are who judged the wines. Personally, I think one judge is best - else you have exactly what here occur; few high scores because the other person didn’t like it. Then again, few high scores make sense, as they’re scoring Industrial Wines. My best advice is to Judge Each Wine Yourself! Trust your own tastes…you do for food, cds, etc., do it for wine, too! Jack, www.ForkandBottle.comBy zumzum
November 3, 2006 8:09 AM | Link to this
Hmmmmm, Bill….exactly what are you trying to say,hmmmmmm? Is there a point?? Hmmmmmmmm??? I am starting to find the whole wine biz (and buzz) about as ridiculous as fortune telling. It is a BEVERAGE people…you aren’t going to get knighted for your vast….hmmmm….saviore faire?? laissez faire?? ohio state fair?? attitudes. When it comes right down to making recommendations that make sense, are priced right, taste good, and right at the heart of what I am looking for…screw Robert Parker et al, and go with Doug at Arrow Wine. He’s my wine hero!By bill
November 2, 2006 11:10 AM | Link to this
Frankly, I’d hoped for a little more than your mousy “hmmmm”s. Suspicious? Got an accusation to make? Want to allege wrong-doing or incompetence? then out with it, in plain English, sir, and none of your p***y-footing “Hmmmm”s. The comments by most of your readers, many of whom reek of incipient wine snobbery, are no better, save for those by the wine-maker and wine-retailer. Let them ask themselves why they are so weak-minded as to be reading and following wine ratings in the first place.By hugh
November 2, 2006 9:13 AM | Link to this
The best thing about the CR report is that it sends the 100 point scale to the graveyard. You would be surprised to find that we wine lovers want only a poor, good, exelent description. I would challange any wine taster to get within 3 points tasting the same wine “double blind” at two different times of the year.By Tony
November 2, 2006 7:53 AM | Link to this
I did’nt see the article. Did it include wines from around the United States or was it just west coast?By Scott
November 2, 2006 7:40 AM | Link to this
As a wine retailer, with fifteen years experience selling great wines from all over the world, I have my doubts about the validity of the CR ratings. While they do pick good value wines each year, they rarely find any of the true gems out there. Many of the best values aren’t going to be mass produced bottlings (such as the wines imported by Eric Solomon and Robert Kacher), so CR won’t rate them. As for which publication we can trust, I tell my customers to find a reviewer that rates on a scale which matches their own palate - some people like the same styles of wines as Parker, while others trust Laube; personally, I find Stephen Tanzer the most consistent. Since CR won’t divulge the identity of it’s tasters, and we don’t know if they use the same tasters each year, how can we trust them to be consistent? I’d have to say that CR is a good source for the wine novice, and will suggest some good values, but if you’re looking for really exciting wines, find a reviewer you can trust, and more importantly, a retailer that will come to know your likes and dislikes, and select wines for you that match your personal tastes.By Larry
November 1, 2006 11:49 PM | Link to this
I used to be a marathoner and I remember a shoe survey that CR posted. I will not say which shoe they said was the best but it only lasted me two weeks!! If you believe anything that they say, you probably voted for George.By Daniel
November 1, 2006 3:44 PM | Link to this
I am the marketing director for a small winery in Napa and I guess I am missing the point on your criticism of CR. They (CR )buy the wines they taste (parker and laube receive samples which they don’t pay for)they have no investment in the industry (last time I checked ws is a paid publication)they have 2 experienced tasters who evaluate the wine but you seem concerned that you don’t know who they are and grant them less credibility than laube or a lawyer? So, you know them? I do. As you are fond of saying, hmmm. I consider any of the ratings systems suspect and your criticism of them more so. Come on floks, I’ve been in this business for 30 years. I’ve been a wine judge, taught wine at 3 universities, made wine on 4 continents, I am a wine judge, a wine writer, planted it, harvested it, pumped it, tasted it and I have never seen laube, parker, or the author on a crushpad. Buy what you like. Try new things. Find a wine store clerk who is not pretentious and take their advice. Sometimes. Enjoy and relax. CR is more credible than so many critics. Its just grape juice.By swillparty
November 1, 2006 1:23 PM | Link to this
CR’s criteria for choosing wines is indeed interesting. Why not venture into the unknown a bit rather than re-rating wines that have “done well” previously? Two mystery tasters do not inspire great confidence in CR’s “objective” ratings. Our SWILL wine tasting club does use Parker as well as other sites for our wine rating research, but I’m not sure we’ll put a lot of weight on CR’s ratings. We’ll see if they have rated anything we are tasting in future gatherings.By Tom
November 1, 2006 1:16 PM | Link to this
Are the ratings at other magazines bought and paid for? It makes you wonder if they rate wines made by their advertisers, doesn’t it? Do their reviewers purchase the wines or receive them for free? And can we find out easily? At least CR puts everything up front.By Cathy
November 1, 2006 12:16 PM | Link to this
Consumer Reports rates wine? I didn’t know that. I suspect the people who consult CR for wine Ratings are non- oenophiles and get what they deserve. Well, that’s not nice… I take it back. Heidi is right to tell her customers to branch out and try less boring wines. I think it’s funny as h3ll that they have a staff of 2 mysterious “wine experts.”By Michael Sarro
November 1, 2006 11:53 AM | Link to this
i agree with consumer reports on their choice of wines based on their criteria for choosing wines. yes, they are safe, boring wines that would never be cellared by a collector, but they wouldn’t be refused in a glass by a collector either. the wine industry should be thankful that consumer reports rates wine. that hits a market that normally wouldn’t be aware of wine at all.By Dave
November 1, 2006 11:50 AM | Link to this
Sounds like they are rating wines that are just entering the “Premium” market segment, $15 and up. Since the wines they are rating seem to be in the $10 - $20 range I don’t think you will find many wines that would make it to the 90pt + mark on other rating scales. The benefit is that the CS report will provide people new to wine with a guide to purchasing good value wine and for those already indoctrinated a list of daily drinkers….By Jill
November 1, 2006 11:20 AM | Link to this
CR is my bible on most things. Wine - the only way to know is to taste it yourself.By Heidi
November 1, 2006 10:51 AM | Link to this
I work in a wine store and I always dread when the Consumer Reports wine ratings come out. I tell my customers that Consumer Reports is good for when you want to buy a car or an appliance, but their wine ratings will only get you good, safe, and boring wines at best.By david
November 1, 2006 10:36 AM | Link to this
Okay, so now we know which boring, mass-produced wine is best. Great!By mel
November 1, 2006 9:54 AM | Link to this
I am not sure how much faith i would put in a publication whose primary job is rating microwaves and washers. However how much faith do you put in any of the wine raters? The only palate that counts is YOURS. If you like it it’s good!!By Stuckdog
November 1, 2006 9:51 AM | Link to this
I have always been suspicious of the Wine Spectator ratings, but what else do you have. I would like to have three or four ratings to compare, but that is rarely the case.By John
November 1, 2006 9:12 AM | Link to this
Consumer Reports has generally done a good job rating wines in the past, at least according to my palate. Really it all comes down to the one doing the tasting. Frankly, Bob Parker has given high ratings to some wines I wouldn’t give to my dog, if I had one.