Understanding the Ultimate Spirits Challenge Scores

Ahead of the 2021 Ultimate Spirits Challenge ratings release, we sat down with F. Paul Pacult to dive into the work behind the scenes to make it happen. Paul is a founding member of Ultimate Beverage Challenge, along with Sue Woodley and David Talbot, and is the judging chairman of this spirits competition company. He is also an award-winning writer, educator, and reviewer who has recently added two new books to his credit, The New Kindred Spirits and Buffalo, Barrels, & Bourbon.

HOW DID THE ULTIMATE SPIRITS CHALLENGE GET ITS START?

Prior to 2010, when we (my wife, Sue Woodley, and me) partnered with David Talbot, I had been involved with beverage alcohol for over 30 years. For the preceding 20 years, I had been writing about and critiquing spirits. I have also been a judge at numerous spirits competitions around the world.

A year after leaving all spirits competitions because we did not agree with their methodology, around the beginning of 2009, Sue and I met with David, and we all agreed that we wanted to create a spirits competition with a fresh approach to judging. It would be a contest that would not involve handing out medals just for entering the competition. The idea was to develop a different variety of scoring methodologies with an approach to bestow awards that recognized the skill of distillation and maturation.

As we spoke with a lot of our friends in the industry, we asked what they would want from a competition. And it kept coming back to me from executives, marketing people, and PR firms saying that what they wanted most was useful information. "I want information on (1) how does my spirit rate within its category against its direct peers (2) what can I do to improve it in case you know, and (3) how can I market it using a competition's results." So we gathered all this information over about six months and came up with the concept that would become the Ultimate Beverage Challenge, the umbrella company that would host wine and spirit competitions. We're now focused solely on spirits.

WHAT IS THE METHODOLOGY BEHIND THIS COMPETITION?

Ultimate Spirits Challenge delivers a unique analytical methodology that is comprehensive and unassailable. It focuses on giving submitters of spirits products information they could use, as a brand ambassador or a marketing firm, to promote their brands. So, it all came down to creating a judging system that was different from other competitions. I'm always tinkering with our methodology because I'm the type of person who is never satisfied with what I do.

Frankly, I don't care if the product is $11.99 or $1,199. What it comes down to in our judging is all about what's in the glass rather than how much it costs. Would I recommend this spirit to a friend? And if I would, to what degree would I recommend it? So, at that point, we decided on a judging scale, setting our sights on the 100-point system. The first reviewer who utilized this system to any effect really was wine critic Bob Parker back in the 1980s. It's a system that has become widely accepted. I think people understand it. From the retail standpoint, retailers can talk to their sales staff and say, "yeah, this got a 94, and a strong recommendation in Ultimate Spirits Challenge".

This delivers a scale of reference that means something to the customer. And the same with marketers. Both, I think, understand the 100-point system very well.

HOW IS EACH SPIRIT JUDGED?

Credibility is everything in a competition. So, I made certain that in USC, we don't just do blind tasting; we do a double-blind tasting. So what exactly do I mean by that? I mean that not only do the judges not know what they're tasting, but the people in our back-of-house crew don't even know what they're bringing out to the judges. The judging room is totally separated from the preparation room. Only two or three people and I know what is being judged, so there's no chance that the judges will ever know what they're tasting. Any awareness of the brand, location, etc., has the possibility of influencing scores, so it is these potentially nuanced influences that we've removed in this double-blind process.

Great people are also key. I mean, I consider myself a pretty decent taster after so long. USC's approach to methodology includes attracting the foremost educators, authors, and journalists in the world to serve as our spirits judges. And we attract those people because we treat them with respect.

We need to be sure not to overburden the judges by evaluating too many spirits in a single day. So, I made the promise to the judges that no one would ever have to taste more than they were able to; flights would be kept small and manageable. There'd be plenty of time in between flights so that they could check email, check their texts, and they'd never judge more than 50 spirits in a day. And frankly, most of it now is about 40 to 45 a day, a manageable amount for a professional taster.

If the judges are fully engaged, if they're fresh, if they're alert, they're going to be rendering better, more accurate results. It just stands to reason to me. And we make certain that every product that's entered goes through multiple panels to give it a level playing field. It's vital to give respect to all the brands that enter. This is accomplished through our meticulous attention to detail in our methodology. Once all the judging is completed, we then compile the scores to arrive at the final aggregate score.

HOW DO BRANDS ENTER THE COMPETITION?

Actually, entering is easy. I think if people look at our website, they will find that it's very, very clear as to how to enter; we go step by step. They just need to look at the codes and classifications for what their particular spirit is. Fill in the order form. And just send it off.

Early on, when Sue, Dave, and I started Ultimate Spirits Challenge back in 2010, we created a content management system basically from scratch with the support of a talented coder who built this specific content management system for our needs. We tried to replicate some of the best features of our favorite retail websites and other online services just to get tips from them on how to do it to make it easy.

HOW SHOULD READERS VIEW THESE SCORES?

I know that with the 100-point scale, retailers often may say that they don't like to have anything in stock that's less than 90 points. But I think that's kind of missing the boat because my feeling is there are so many good products that scored 85 to 89. And often, they're very nicely priced, so you have a good price-quality ratio there.

• 85 to 89 is considered good and of substantial quality. I think these are very respectable scores.
• From 90 to 94, we consider those as something extraordinary. This range is better than average, showing the kind of quality that we know a lot of producers strive for. These are top-flight products that are well worth getting in your car and looking for.
• Once you get to 95 to 100, that's the best of the best. Those are products that, to me, I'd call milestones of that category. They set a benchmark that reflects what's the best for that segment.

We have only had two 100 scores over the last 12 years. So, our judges are not easy. And I take pride in that. This is a contest to ascertain quality so the judging should be difficult.

ASIDE FROM SCORES, WHAT OTHER DESIGNATIONS CAN SPIRITS RECEIVE?

The Chairman's Trophy, of course, is the pinnacle recognition of excellence within every category. We only give 45 to 55 Chairman's Trophies a year. And we also recognize those spirits that were in the running for the Chairman's Trophy with the Finalist Award because we've heard from marketers and brand managers that this designation is also important.

The Tried and True Award is something that we instituted a few years after 2010; it came in about the third or fourth year. We were beginning to see that products that were entering year after year were showing high levels of consistent quality. I think that's something that shows the consistent quality of a spirit. And that should be rewarded.

As we've evolved as a competition, I always go back to the Great Value Award as my personal favorite because I love a bargain. There are so many fabulous spirits right now that are affordable, products that won't hurt anybody's pocketbook. And yet offer a remarkable level of quality for that category.

And we recently instituted the USC Top 100 Spirits award for the top 100 spirits for the Challenge year.

WHEN DO YOU INCLUDE TASTING NOTES WITH YOUR RATINGS?

When we asked a lot of key executives in the alcoholic beverage industry what they wanted from a competition, one of their points that kept coming back was, "Give us tasting notes." It was this request to give them the language that they can utilize to promote their products that convinced us that detailed tasting notes would be helpful.

So, after deciding upon using the 100-point scale, we opted to provide notes for all spirits that received a score of 85 or higher. And it's so pleasing to me to see a shelf talker that has a USC tasting note and a Score of 87.

WHAT IS MOST EXCITING IN THE SPIRITS SPACE RIGHT NOW?

I think mezcal and agave spirits have just continued to expand and show the glory that is created when certain agave is fermented and distilled. They occasionally show extraordinary terroir. You can sometimes pick up the variety. Maybe it's Tobala, or maybe it's Arroqueno, but I think there's so much growth right now in agave spirits, and the quality across the board overall is exciting.

We've also seen that the Ready To Drink (RTD) category is growing substantially. We had around 108 to judge this year; last year, there were just 30 or 40.

Even non-alcoholic beverages are showing up in greater quantities. We've had more entries this year than probably the previous 11 years. For these, what we did was we had our back-of-house team mix them up as cocktails to see how they would react in a mixed situation.

And one more, gin. You know, just when we started out in 2010, maybe we had 10 or 12 gins entered. This year, it's well over 100. So, that category has grown dramatically. It's remarkable. And the gin space is so interesting because many distillers who are producing gin are doing so with more floral and vegetal influences, rose petals, violet, chamomile, and cucumber that are adding new dimensions to gin. Meanwhile, other distillers are going "retro" by returning to the older, traditional styles, such as "Old Tom" gins.

ANY CLOSING THOUGHTS?

Even though the spirits industry views USC as the platinum standard of competitions, we are never going to be satisfied with how we do things. I always feel we can be better. I would never like to think that we're gonna rest on our laurels. Never.

It all comes down to respecting the distillers of the world, first of all, because I think they're remarkable men and women. And that's why we're so crazy about our methodology. And if people think we're over-the-top about our methodology, that's the greatest compliment. Because I think the world's distillers deserve that; they deserve everything that we could possibly do, from a competition standpoint, to honor what they do and show the respect to their products that they deserve. And I hope that more consumers will come to recognize what makes Ultimate Spirits Challenge different.

EDITOR'S NOTE

There are a number of wine and spirits competitions, such as the Ultimate Spirits Challenge that work diligently to inform brands and consumers about the quality and impact of each submission. As you look to explore a new spirit or brand or discover a new bottle that you find interesting, the resulting scores, results, and tasting notes offered by these competitions can provide you with useful details to help inform your purchases. They can also offer you a jumping-off point for identifying your own tasting notes and consideration about the scores given.

The new 2021 Ultimate Spirits Challenge ratings have been released, and you can find them here right now.