Action Plan for Diagnosing Product Health

Summary

The website content outlines a systematic approach for diagnosing changes in product metrics, emphasizing the importance of investigating data quality, product changes, behavioral changes, and mix shift.

Abstract

The final post in a seven-part series on analyzing metric changes, this article provides best practices for diagnosing product health. It advises selecting two time points to investigate metric changes, considering factors such as product changes, seasonality, competition, mix shift, and data quality. The process involves generating hypotheses, eliminating factors, and quantifying impacts. Data quality issues should be addressed first due to their ease of identification. Product changes are scrutinized by listing alterations, quantifying impacts through experimentation frameworks, and examining behavioral changes by group. Seasonality and external events are acknowledged as significant contributors to behavioral changes. Mix shift is considered, especially for long-term metric changes. The article concludes that metric shifts are multifactorial and challenging to diagnose.

Opinions

Data quality issues are considered the easiest to identify and should be investigated first.
Product changes require careful examination, including the use of A/B testing to quantify their impact on key metrics.
Behavioral changes due to seasonality, external events, and competition are recognized as important factors in metric fluctuations.
Mix shift is seen as a potential cause of long-term metric changes but is less likely to be the source of week-over-week changes.
The article emphasizes that shifts in metrics are complex and usually result from a combination of factors, making accurate diagnosis challenging.

Action Plan for Diagnosing Product Health

In this final post of our seven-part series on analyzing metric changes, we offer additional best practices for addressing metric changes due to the factors discussed in previous posts: product changes, seasonality and other behavioral changes, mix shift and data quality.

Once you have confirmed that there is indeed a change in metric worth investigating, you need to develop a systematic and structured approach to identifying and attempting to eliminate each possible cause.

The first step is selecting two points in time that best represent the change in metric you are investigating. (As explained in Part 4, the larger the change and shorter the time frame, the easier it will be to identify the root cause.) Next, you should ask lots of questions about what could have caused the change in your key metric. Once you have a comprehensive list of hypotheses, eliminate or investigate factors one by one:

DATA QUALITY

Investigate issues with data quality first, as they may be the easiest to identify. Look for logging issues related to product changes — for example, a bug that incorrectly records DAU (daily active users) for a certain locale, language, country, device, etc.

To localize such an issue, investigate whether the change is systemic across all dimensions or specific to some dimensions.

Examine other correlated metrics for similar changes. For example, if number of sessions is correlated with DAU, and you see a change in DAU but not number of sessions, a logging bug may be the problem.

Refer to Part 6 for details on data quality issues.

PRODUCT CHANGES

List changes made to the product in the given timeframe. (If no changes were made, you can eliminate this factor, but note it’s possible to forget. Find a way to account for this by tracking changes.)

If you have an experimentation framework (A/B testing), quantify the impact each product change would have on your key metric.

Look for behavioral changes due to product changes. Examine behavioral changes by group (country, device, etc.) to determine whether the changes are localized, then examine the time at which you saw the metric change. If the metric change happened outside of the time period you would expect based on the timing of the product change, it is unlikely the latter caused the former. Remember, too, that network effects can sometimes carry the impact of an issue beyond the population that is primarily affected; for example, if a bug prevents people in Israel from using a communication platform, this could lower engagement of people in other countries, as well.

MIX SHIFT

As explained in Part 5, the first step to diagnosing mix shift is hypothesizing the dimension in which you expect it to occur. Part 5 offers specific guidance on quantifying the effect of mix shift. Note mix shift may be a strong factor in long-term changes, but likely will not be the source of changes week over week.

TAKEAWAYS

Shifts in metrics are almost always due to data quality, product changes, behavioral changes and/or mix shift.

Because shifts are rarely due to just one factor, they can be difficult to diagnose.

This work is a product of Sequoia Capital’s Data Science team. Jamie Cuffe, Avanika Narayan, Chandra Narayanan, Hem Wadhar and Jenny Wang contributed to this post. Please email [email protected] with questions, comments and other feedback.

Action Plan for Diagnosing Product Health

DATA QUALITY

PRODUCT CHANGES

BEHAVIORAL CHANGES

MIX SHIFT

TAKEAWAYS

This story is published in The Startup, Medium’s largest entrepreneurship publication followed by 361,652+ people.

Subscribe to receive our top stories here.