Bad Assumptions Come From Bad Data.
Make Sure You’re Measuring the Right Things.
We all want to understand how (well) things work, whether they are processes, activities, or systems. We need to measure their performance, quality, or outcome using some indicators, which we call metrics.
We also need to collect, analyze, and interpret the data using some approaches or frameworks, which we call methodologies.
Metrics and methodologies are important tools for making assumptions, which are educated guesses based on the evidence we have.
However, not all tools are created equal. For different situations, some metrics or methods are more valid, reliable, or relevant than others.
If we use the wrong tools, we might make assumptions that are wrong, false, or biased. This can make it hard to make good choices, solve problems, or come up with new ideas.
It is important to use metrics and methodologies that align with our goals, objectives, and context.
A bad example
A man who drank 3 liters of water per day, died at the age of 95.
Conclusion: Drinking over 100000 liters of water is lethal.
This is a ridiculous hypothesis, but it shows how using the wrong metrics (the amount of water drank) and the wrong methodology (completely ignoring other factors) can lead to a false conclusion (drinking water is lethal).
The metric of water consumption is not a reliable indicator of the cause of death, as it fails to consider many other factors that impact human health and mortality.
This example demonstrates the importance of using the right metrics and methodologies to make assumptions, as using the wrong ones can lead to wrong, irrational, or harmful conclusions.
More examples of bad metrics
Hours spent in the office
A popular metric for measuring employee productivity.
This is a common bad metric, as it has been used all over the world for decades. Businesses pay their workers based on the hours spent working for them, but not every hour is the same. Time spent in the office does not directly correlate with productivity.
This perspective fails to consider the varying work styles and capacities of individual employees. Some workers can do more in a focused four-hour period than they do in an eight-hour day.
The focus on hours worked doesn't consider the importance of efficiency, innovation, and creativity, which are often hurt by long, rigid hours.
Number of followers on social media
Likes ain’t cash
Internet “influencers” have the most distorted perspectives when it comes to the impact the number of followers actually has on their “influence”.
There are numerous stories about “influencers” trying to sleep in expensive hotels “for free” in exchange for an Instagram post or eating in a fancy restaurant in exchange for a blog post.
There is one story about a girl called “Bhad Bhabie”, who quickly grew her social media following to millions of users. A couple of months later, she announced she would be launching her clothing line.
The products were overpriced and unoriginal. The followers didn’t convert to customers, the clothing line was a commercial failure.
An excellent example of “likes are not cash”.
Number of users
Focusing only on the number of users, ignoring other factors around the product
Did you know Apple nearly killed the iPod because there weren’t enough users?
When the iPod was first released, it was not very popular. Luckily for them, they decided to improve user-friendliness. They made it smaller and easier to use. Their 4th generation of music players made them popular.
Netflix, when they were still renting DVDs, was not very popular. The company was losing money, and they considered killing the service. They decided to focus on making it more convenient for customers and launched a service where customers could order DVDs online and get them delivered to their homes. Netflix is today one of the most popular companies in the world.
These are just some cases where businesses could have made a wrong decision if they only focused on a single metric
How to do this right?
Here are some tips for choosing the right tools for evaluating anything:
- Define the metrics that are specific, measurable, achievable, relevant, and time-based (SMART) for our goal.
Example: To see if a new medicine works well, we can use the percentage of people who get better after taking it for a certain amount of time.
- Choose methodologies that are appropriate, rigorous, and transparent for our data.
Example: If we're looking to assess the impact of a fresh regulation, we might use the approach of a randomized controlled trial. In a randomized controlled trial, we compare the outcomes of a group of people who've been exposed to the regulation with those who haven't while adjusting for any other variables that could affect the outcomes.
- Test the assumptions that are logical, consistent, and falsifiable for our evidence.
Example: If we want to test the assumption that drinking water is lethal, we might look for counterexamples, such as people who drink water and live longer than 95 years, or people who don’t drink water and die sooner than 95 years.
By doing so, we can ensure that our assumptions are based on the right metrics and methodologies and that they are valid, reliable, and relevant for our situation.
These skills will help us improve our understanding, insight, and innovation in any field.
