The Premium Metric


tldr;


The Premium Metric was created to measure if a brand or product feels like a premium experience to customers/users. Developed using over a dozen studies with 500+ individual test takers in multiple markets, the Premium Metric is a tool to provide simple and immediate feedback on the perceived premium-ness of an interface, feature or product.

Take me to the why.

The Challenge

As a product designer it’s not uncommon to hear stakeholders voice their grand visions for their upcoming offerings in jargon and manager speak. They have ideas of how the customer will describe it and what qualities it will take to turn potential buys into dollar signs.

“Make it modern” they say, or “make it just like Brand X they’re premium.”

Ok, but what makes it premium? What even is premium?

How do you measure a qualitative attribute besides customer verbatims? Is it possible to compare something so subjective in a way that stakeholders can understand?

My Hypothesis

Premium can be measured with a qualitative methodology, similar to the System Usability Scale, by understanding how users perceive and think of premium. If the commonalities of premium can be understood across a data set then they can be quantified and compared.

Exploration and Metric Creation

TEST SETUP

An initial pool of 20 participants that fit the profile of a core or adjacent luxury vehicle customer, participated in a 90-minute usability study in September, 2017. An additional group was tested from a pool of participants in the United Kingdom. As a part of this initial study, a card sort was completed by each participant to understand how a customer defines the qualities of ‘premium’.

METHODOLOGY

Participants were asked the define the word ‘premium’ and to characterize the qualities thereof. After providing a definition, the participants were presented with 8 individual IVI home screens. Participants were unaware of the screens’ origins and all branding had been removed from the images. The participants were asked to order the screens from most to least premium based on their individual definitions and talk-aloud while performing this task. Following a usability test of a prototype, participants were asked if the system they had interacted with met the premium criteria they had established.

High Level Findings

Initial high level findings were common and consistent across respondents.

  1. Premium is simple, not basic.
  2. Premium puts top priority on driving by surfacing timely and meaningful information.
  3. Premium offers lots of options, but isn’t busy.
  4. Premium thoughtfully balances appearance, ergonomics, usability and design elements.
  5. Premium includes graphics tailored to enhance the information presented.
  6.  Premium is not common.
  7. Premium is the highest level of value. The exchange of value for cost is positive.
  8. Premium exemplifies high quality.
  9. Premium presents content that is informative and robust.
  10. Premium makes features easy to access and available “at my fingertips.”
“Premium means quality. It will be of a good quality and that could be in aesthetics, that could be the quality of the thought, the amount of thought in the investment they put into that.”
- Participant verbatim
CONTINUED UNDERSTANDING

Further studies were completed and from over 1000 responses there were 26 qualities of premium that stood out. These 26 qualities were recognized by the participants while expressing their personal definition of premium and evaluating the prototype. Patterns emerged throughout the process that developed semantic differentials for the qualities, i.e., each item is represented by two terms with opposite meanings.

A premium product would represent as many, if not all, of the positive differentials as possible.

The 26 Qualities of Premium

Above Average

Big

Calming

Thoughtfully Colorful

Connected

Convenient

Craftsmanship

Customizable

Elegant

Exceeds Expectations

Expensive

For me

Visual


Below Average

Smaller (Limiting)

Aggressive (Overloaded)

Monotone

Isolated

Buried

Inferior

Fixed

Crude (clumsy)

Disappointing

Cheap

Mismatched

Text Heavy


High Value

Higher End

Informative

Intuitive

Luxury

Makes you feel good

Minimalist

Modern

Feature Rich

Exclusive

Top of the Line

Top Performing

Visually Appealing


Wasteful (meaningless)

Knockoff

Ambiguous

Difficult

Common

Dissatisfied

Exhaustive

Obsolete

Limited

Generic

Economy

Unreliable

Ugly


“When I think of that word I think of something that's usually offered like a little bit better — an item or object or service that you get that is a little bit above something that people normally get for a type of thing.”
- Participant verbatim
GOALS OF THE METRIC

Once the qualities of ‘premium’ are established, a metric should be defined to accomplish these goals:

  • Provide a simple and efficient way to measure and interpret the premium-ness of an interface, product, or feature.
  • Establish a premium baseline for products to meet or exceed. Score a feature, interface, or product based on the agreed definition and qualities of premium.
  • Potential to create a second metric that evaluates ‘premium’ service.
  • Determine whether a feature or product meets the expectations of something that is premium.
  • Provide signal on any qualities that are lacking or considered ‘failing’.
  • Compare iterations of a feature or product to track progress with respect to premium design/experience over time.

Measuring Premium

The metric provides a quick score to determine the adjacency of premium.

Through ongoing studies the score established as a premium threshold is approximately 78/100.

As more data is collected the threshold is being refined.

Premium Threshold
100/100
Anything lower needs improvement

In addition to the completing the metric, each participant is asked a binary yes or no question as to whether the product or interface evaluated meets their personal definition of premium. This allows the threshold to be evaluated and the overall score provided matched to the user definition.

The below graph shows the results for an interface study completed across three markets, comparing the market results to the overall scores of all data sets.

Image
"Premium means elegant, It means top performing, it means, just um... intuitive, something your needs are met in advance rather than something you have to muddle through. It's already there for you. It's more intuitive to most people."
- Participant verbatim

The score is also broken down into “hot-spots” that allow the researchers and designers to understand where the product or interface performed best and worst.

These hotspot graphs allow for actionable changes to be made to previously nebulous feedback about aesthetics, tone, brand, and expectations.

Two different studies shown before demonstrate the hot-spot results provided.

Image Image

So what’s the big deal?

Previously qualitative data such as whether the right qualities of a product or interface evoked the right tone or brand voice were hard to communicate or understand. It’s very difficult to base competitive qualitative data against a baseline when you don’t understand what you’re looking for.

The Premium Metric removes some of the ambiguity of “I don’t like it, but I don’t know why” or “I love it because it’s so luxurious” in product design – which is especially useful when designing interfaces to match physical products.

I created this metric because I was sick of hearing stakeholders and designers use premium as some buzzword that they could just toss around without proof.

You can’t just say something is premium. Saying it doesn’t make it true.

With the premium metric designers and researchers can take actionable feedback away from a qualitative user study and integrate directly into UI designs and user flows, targeting the right aesthetic qualities and letting things within the product that work remain exceptional.

This metric is about enhancement to usability and understanding qualitative feedback at a core level. It also makes communicating why something is or isn’t premium back to a stakeholder much easier.