Summary

Chef Antonella's new restaurant's success metrics, such as revenue and table reservations, are positive, but Key Experience Indicators (KEIs) reveal issues like high food waste and a lack of repeat customers, prompting the need for focused experience measurements to improve customer satisfaction.

Abstract

The article discusses the importance of measuring Key Experience Indicators (KEIs) to gain insights into specific aspects of a customer's experience, which may not be evident through traditional business metrics. Chef Antonella's restaurant serves as a case study where, despite financial prosperity, KEIs indicate a near-total absence of returning customers and significant food waste. The article emphasizes that KEIs provide actionable data, precede business outcomes, and complement qualitative research. It outlines considerations for KEI measurements, including the distinction between what vs. why, good vs. bad metrics, and the need to prioritize behavior over attitude. The article also advises measuring specific experiences over general ones, using direct instrumentation for analytics, and ensuring that measurements are taken in context. It concludes with the recommendation to only measure aspects of the experience that one intends to change, to gain team consensus on the importance of KEIs, and to focus on core experience metrics.

Opinions

KEIs are crucial for understanding specific areas of the customer experience that need improvement, which traditional business metrics may not capture.
It's important to pair quantitative KEI data with qualitative research to fully understand the reasons behind the metrics.
Not all high scores or long engagement times are positive; they must be interpreted within the context of the overall experience.
Behavioral metrics are more reliable indicators of experience than attitudinal surveys, which can be influenced by external factors.
Measuring overall satisfaction, such as NPS, is less useful than measuring specific elements of the experience to identify areas for improvement.
Companies should instrument their products or services directly if existing analytics tools do not provide the necessary indicators.
Feedback should be collected in the moment or immediately after the experience to ensure it is contextually relevant.
Only metrics for which there is an intention and capability to make changes should be measured.
There must be team agreement

Key Experience Indicators: How to decide what to measure?

When you measure smaller parts of the experience (only important ones, not everything), you get an indication of what specifically works well and what doesn’t. You can then dive deeper into the problem areas to understand why.

Chef Antonella opened her new restaurant six months ago. Everything has been going really well, it seemed. Revenue is up and to the right, tables are reserved in advance, and the restaurant is full and lively every single night. Business metrics show a very positive picture.

Chef Antonella also measures two key experience indicators she cares about. These paint a different picture. Close to 100% of customers are new. She has almost no returning customers. Second, on average, 70% of food is left on the plates. Something is wrong although business is great.

🔑 Why measure experience?

Key Experience Indicators (KEIs) provide a quantitative score of a specific, important, and actionable phenomenon related to using a product or service. Measuring KEIs has the following benefits:

KEIs provide information to decision makers.
KEIs precede or predict business outcomes.
KEIs give insights into qualitative findings and customer anecdotes.

🔬Considerations of KEI measurements

What vs. why: KEIs will never explain themselves. To understand a KEI, one must invest in qualitative research.

Results of experience-related measurements for three alternative designs. Design a is better than b and c, yet you don’t know why until you make an effort to learn.

Good vs. bad: It is not always clear whether KEIs are showing a positive or negative phenomenon.

Can you tell if these are good or bad?

Average time to first click is 49 seconds.
Average time on site is 27:38 minutes.
Homepage bounce rate is 99%.

Attitude vs. behavior: what people say does not necessarily match how they behave. People’s attitude is interesting yet behavior is much more telling. Give more attention and weight to behavioral rather than attitudinal metrics.

Chef Antonella’s restaurant is providing great coffee. Yet if you ask one of Antonella’s customers to rate her satisfaction with coffee immediately after she found a mistake on the check, coffee might be getting low scores. Coffee consumption, however, measured in the same exact time, might tell a more realistic story.

Beware not to measure behavior through attitude. Don’t ask about behavior in a survey. Instead, measure it directly.

Overall vs. specific: Measuring specific phenomena gives you the ability to focus efforts and save resources. When you measure NPS or overall satisfaction, you get scores that do not help you understand what goes well and what needs fixing. If you get a score of 66 for your product’s NPS, do you know why you lost 34 points?

Alternatively, when you measure a lot of smaller experiences, for example, at a hotel, you may find that while your overall UX score is 93%, you lost 7 points because coffee, room cleanliness, and the online checkout process got low scores. You wouldn’t know why and yet you can start exploring these three areas rather than guessing what’s wrong.

Instrumentation vs. analytics: Measuring the user experience sometimes involves instrumenting dedicated sensors within your product or service (physical or digital) because different analytics packages don’t always give you the indicators you need.

In vs. out of context: When measuring attitude (what people think), it’s very easy to fall into the trap of sending a survey once in a while, yet a survey is not “close” to the experience. The best way would be to ask during or immediately after people have a certain experience. Leave surveys to questions that don’t make sense to ask otherwise.

At WeWork, we experimented with measuring attitude toward different parts of the experience. For example, we asked about satisfaction with phone booths and printers in those spaces, about events using an app we created, and left asking about the office chairs to a quarterly survey (chairs don’t change much and it didn’t make sense to put a happiness meter on each one).

🥇Only measure things you have full intent to change

Spoiler: you can change anything in the experience you created. It’s all a matter of motivation to do so. This one is important. Your customers will stop cooperating with your attitudinal measurements and get angry with you if they notice you don’t change the experience after they provide feedback. Test question for the team: Will we take action once we have data for this metric?

🥈Reach team agreement and shared understanding

Agreement within the team and senior stakeholders that a certain KEI is important. Disagreement results in ignoring KEIs and not taking action. Make sure to create a shared understanding and acceptance of what’s being measured. Test question for the team: Do we agree this metric is important to measure?

🥉Only measure things that are core to the experience

Measuring everything is measuring nothing. Focus on measuring things that are truly central to the experience. Together with your team, reach agreement on what these are. You may have some debates here but for the most part, those core things are going to be easy to identify. Test question for the team: Does this metric measure a thing that is core to the experience?

🎯 In a separate series of articles, I share a set of useful KEIs to measure based on the Google HEART framework: Measuring user happiness Measuring user engagement Measuring user adoption Measuring user retention Measuring user task success

Meanwhile, please 👏👏👏, share, and comment. I’d ❤️ to hear from you. How difficult (or easy) is it for you and your team to decide what to measure?

If you liked this article, check out:

Measuring User Happiness eBook.
Measuring User Happiness is Product Management podcast episode.

You can also find me on Twitter if you’d like to follow more of my research thinking!