avatarDeepak Chopra | Talking Data Science

Summary

The web content discusses the concept of hypothesis testing in the context of marketing measurement, emphasizing its role in determining the effectiveness of marketing interventions through experimental design and AB testing.

Abstract

The article delves into the statistical method of hypothesis testing as a cornerstone of marketing measurement. It outlines the process of constructing a test-control analysis to assess the impact of marketing interventions on consumer behavior. The approach involves dividing the audience into two similar groups, exposing one to marketing stimuli while withholding it from the other. The core assumption is that any significant differences in behavior post-marketing exposure can be attributed to the intervention itself. The article defines the Null Hypothesis (H0) as the belief that there is no difference between the two groups due to marketing, and the Alternative Hypothesis (H1) as the belief that there is a significant difference. It also touches upon the importance of ensuring no confounding factors skew the results and the necessity of observing non-trivial differences to reject the Null Hypothesis. The process is illustrated with an example of a marketing campaign for a new iPhone, where sales figures between the test and control groups are compared to evaluate the campaign's effectiveness.

Opinions

  • The author posits that marketing measurement is crucial for proving the value of marketing efforts and that experimental design, AB testing, or test-control analysis is the best method for this purpose.
  • It is emphasized that the two groups in an experiment must be initially similar and free from confounding factors to accurately attribute any observed behavioral differences to the marketing intervention.
  • The article suggests that numerical differences observed between groups may often be trivial or by chance, and it is essential to distinguish these from significant differences caused by marketing efforts.
  • The author notes that hypothesis testing involves making decisions about unknown entities, such as marketing effectiveness, based on observed data, and that this process is fundamental to robust marketing measurement.
  • The article implies that statistical significance is critical in determining whether observed differences are due to chance or the marketing intervention, with a promise to elaborate on this in a subsequent post.
  • The author encourages readers to connect, learn, and grow by following them on various platforms and considering Medium membership, hinting at the value of continuous learning and professional growth in the field of marketing analytics.

Marketing Measurement-2: Understanding Hypothesis Testing

Understanding the Construction and Process of Hypothesis Testing in Measuring Marketing Interventions

Photo by Hans Reniers on Unsplash

Marketing measurement is a critical topic discussed in organizations worldwide. Marketers face the ever-important task of proving the value of their efforts, often backed by millions of dollars in budgets. Meanwhile, marketing analysts have the enviable job of uncovering the true impact of these programs.

As we explored in Part 1 of this article, experimental design, AB testing, or test-control analysis stands out as the best-in-class methodology for measuring marketing interventions.

→ Incrementality = Cause-and-Effect

The essence of marketing impact lies in its direct consequence on consumer behaviour. This impact can be determined through a test-control analysis, wherein the audience is divided into two mutually exclusive groups that are similar in all aspects. One group is exposed to marketing interventions while the other is held back as the control group.

The basic premise is that both groups are initially similar, and no confounding factor disproportionately affects either group during our analysis period. Therefore, any non-trivial behavioural differences observed post-marketing exposure can be attributed to the marketing intervention.

Let’s double-click on the above statement — it is composed of three parts:

1. “The basic premise is that both groups are initially similar”.

We start with the underlying assumption that there are no pre-existing differences between the two groups. If the marketing intervention is ineffective, it should not lead to any difference between the two groups post-exposure. → This implies that in case of ineffective marketing, there would remain no difference between the two groups post-marketing exposure

This basic neutral hypothesis of no difference between two groups (or no impact of marketing), is called Null Hypothesis, represented by H0. — The alternative to this would be the hypothesis of observing an actual non-random difference between two groups (or the marketing being effective at driving behaviour differences) is called the Alternative Hypothesis (H1).

NOTE: The difference in the behaviour of two groups is measured quantitatively and numbers being numbers would always have some trivial, random or by-chance difference. We acknowledge that numerical differences will always exist due to chance, but if they are trivial or by chance, we consider them as no difference.

2. “..no confounding factor disproportionately affects either group during our analysis period”

This extension of the first part ensures that the two groups remain similar during the marketing experiment as well. No factor should disproportionately affect one group more than the other. → This ensures that the only difference between the two similar groups lies in the presence (and absence) of exposure to the marketing intervention.

3. “..any non-trivial behavioural differences observed post-marketing exposure can be attributed to the marketing intervention”.

In contrast to the previous argument, if the marketing intervention is indeed effective, it should induce a change in the behavior of the group exposed to marketing, while no change occurs in the group not exposed to marketing. This will result in a difference between the behaviours of the two groups. → Essentially, if the marketing intervention is ‘effective,’ we should observe some non-trivial / non-by-chance / non-random differences between the behaviours of the two groups.

Please note that in order to reject the Null Hypothesis, we must observe some non-trivial/non-random/non-chance differences between the group exposed to marketing (test) vs. the group not exposed to marketing (control).

This observed difference between test vs. control is classified as non-by-chance or non-trivial if it exceeds a predetermined threshold and is often labelled as a ‘significant’ difference. — (more on this in the next post)

→ Comparing the observed difference to a pre-existing confidence level would help us reject or not-reject the Null Hypothesis.

NOTE: Based on the observed data, we are either rejecting or not-rejecting the Null Hypothesis (H0); in any case, we are NOT accepting the H0. — That is because we are observing one particular scenario (i.e. sample) based on which we have evidence to support or not-support our H0; there can be other scenarios (i.e. samples) which might convey something entirely different

This process of forming hypotheses and evaluating them based on observed data is known as hypothesis testing.

Hypothesis testing:

Hypothesis testing is a statistical method used to make decisions about an unknown entity, such as marketing effectiveness or differences in behaviour, based on observed data from known entities. For example, we can observe the actual behaviour of a sample comprising two groups post-marketing exposure.

The hypothesis testing process involves assuming a neutral H0 and seeking evidence in the observed data to either ‘reject’ or ‘not reject’ H0.

Assume H0 → observe the sample data →Reject or Not Reject H0

Understanding with an Example:

Let’s continue with the example we used in Part 1: Apple running a marketing intervention (ads) to promote their new iPhone to a small group of people

We design an AB test (experiment) to measure the impact and randomly divide our subjects into two equal groups: Group A, exposed to the ad (the test group), and Group B, not shown the ad (the control group). For simplicity, let’s assume both Group A and Group B consist of 1000 people each.

In this instance, our hypothesis testing framework comes into play.

  • Since Group A and Group B comprise people from the same population, and the assignment to either group is random, we have no reason to believe there are significant pre-existing differences between the two groups. → the two groups are similar before the campaign.
  • During our analysis period; that is, during the marketing -intervention there are no other factors (for example, any other marketing campaign) which effects either group more or less than the other. → no confounding factors impact either group during the activity.
  • Additionally, we will examine and compare the post-campaign behaviour of Group A and Group B. → As there are no initial differences between the two groups, any significant (non-chance) differences observed post-marketing can be directly attributed to the marketing intervention.

In this scenario, we consider “sales” as the best indicator of buying behaviour. Thus, when we study the difference in behaviour between the test and control groups, we are essentially examining and comparing the differences in iPhone sales between the two groups.

Forming Hypothesis:

Our Null Hypothesis would state that there is no impact of marketing, indicating no significant difference in test group behaviour compared to control group behaviour post-marketing. H0: Test_Sales = Control_Sales

On the other hand, our alternative hypothesis suggests that marketing had an impact, implying that the behaviour of the test group is significantly different from that of the control group. H1: Test_Sales != Control_Sales

Summary

Robust marketing measurement is achieved through an experimental design (AB testing) framework, which involves dividing the target audience into two similar sub-groups: (1) Test: which gets exposed to marketing intervention. (2) Control: which gets suppressed and is not exposed to marketing intervention.

  • The two groups are similar, to begin with in every aspect other than marketing exposure.
  • During the intervention, no factors disproportionately affect either group; resulting in the two groups remaining similar throughout (other than the exposure and non-exposure to marketing intervention).
  • Any non-by-chance behavioural differences that arise post-marketing exposure in the two groups area direct consequence of the exposure to the marketing.

Hypothesis testing is a statistical method used to make decisions about marketing effectiveness or differences in behaviour based on observed data. It involves assuming a neutral Null Hypothesis (H0) and evaluating evidence in the observed data to either reject or not reject H0.

The Null Hypothesis (H0) is the neutral hypothesis that assumes the status quo (of no relationship or no difference) in the population data. H1 is the alternative to H0 and is called the alternate hypothesis.

Statistical significance is a measure of the probability that an observed difference between two groups is due to chance - to be elaborated in the next article

Connect, Learn & Grow ..

If you like this article and are interested in similar ones follow me on Medium, LinkedIn, connect with me 1:1, join my email list and (..if you already are not..) hop on to become a member of the Medium family to get access to thousands of helpful articles. (I will get ~50% of your membership fees if you use the above link)

.. Keep learning and keep growing!

Data Science
Marketing Analytics
Hypothesis Testing
Marketing Measurement
Data Analysis
Recommended from ReadMedium
avatarKseniia Baidina
A Guide on Estimating Long-Term Effects in A/B Tests

Addressing the complexity of identifying and measuring long-term effects in online experiments

9 min read
avatarDeepak Chopra | Talking Data Science
Decoding Statistical Significance: A Marketer’s Guide

Marketing Measurement-3: A layman’s guide to understand statistical significance while measuring marketing effectiveness.

10 min read
avatarAndrii Shchur
Enhancing A/B Testing with Pingouin: A Python Statistical Powerhouse

Unlock the full potential of A/B testing with concise, precise statistical analysis using Python’s Pingouin library.

7 min read
avatarGreg Hopkins
Dealing with Fire Insurance in Italy

Some harsh lessons learned after a fire in the house

8 min read
avatarRosalyn Liu
Lift Study vs. A/B Test — Do You Know the Difference?

A/B Tests Falling Short? Learn About Lift Studies — The Powerful Technique Data Experts Rely On

4 min read
avatarJohanna Jagow
How to manage research projects without losing your mind

Move your research project management to the next level with these five best practices.

5 min read