8 Reasons Why Your Ad Attribution Approach Is Wrong

The problem with the Organic Ratio and how to track the impact of untrackable users instead

This article breaks down all the shortcomings of Organic Ratios when analyzing the performance of digital advertising campaigns, and then describes two step-by-step ways to solve the problems.

A spectre is haunting the ad spend world — the spectre of the Organic Ratio. But the powers in charge of ad spend attribution are oblivious to its dangers. It’s time we exorcise it from digital advertising departments and media agencies, from the brains of marketing VPs and ad campaign managers.

If your company is a heavy spender of digital ads, you might know what I’m talking about. Every day, thousands of companies spend billions of dollars without knowing their impact! They think they do. Somewhat. But they have no idea. You wouldn’t believe the horror stories I’ve seen.

One company I know lost tens of millions of dollars buying ad impressions they thought made them money, but didn’t.

Another company spent so freely that it burned all the money from its investors and had to fold.

Yet another one didn’t realize its ad spend was making a hefty profit, so it ended up closing a business unit that was profitable without realizing it.

Each one of these stories is a drama. All of them together are a tragedy.

When you don’t know the impact of your ad spend, it’s because your attribution is wrong: You can’t attribute the right business impact — like users acquired, or their revenue — to the right spend — like an ad campaign on Facebook.

The organic ratio is one of the main causes of that. It’s defined as how many users you get for free each time you pay for ads to get a customer. Most companies calculate it poorly, and it is one of the costliest mistakes they make.

Let’s take an example.

Imagine that you get 100 new users per day before you start a digital ads campaign (or “PAC” for paid acquisition). Then, you start your PAC campaign and boost total acquisition to 350 new users per day, of which you can track 100 directly back to the ads. That means you are getting the previous 100, the new 100 directly from ads, and 150 new users show up for free. How is that possible? You were just getting 100 before and you only paid for 100 more. You didn’t pay for the new 150. This is an “organic ratio” of 1.5: for every user you pay for, you get 1.5 for free.

You can see an example on the graph above: During the campaign, you track your ad acquisitions, count the number of organics during that time, and attribute those to the ads.

That ratio is very important if you spend a lot on ads. In this case, imagine every new user brings to the company $200 in revenue, and it costs $200 to get new users through ads. You’d think you only break even. You would probably not scale the advertising spend, and thus might end up not growing the company at all if PAC is your main growth channel.

However, if you have an organic ratio of 1.5, suddenly every $200 in advertising cost brings you $500 in revenue ($200 from new users directly from ads and $300 from the organic ratio). You now have a money-making machine! You’re ready to raise millions from VCs (venture capital firms) to scale up spending, or ask for a massive budget increase for the next cycle.

However, just by looking at the graph, you can start seeing all the ways this can go wrong. There are some obvious mistakes — which, thankfully, most marketers avoid — but some are painfully frequent. Here are the top reasons why this organic ratio doesn’t work:

1. It Doesn’t Always Account for the Baseline

Some users were already being acquired organically before the campaign. The least you can do is take these out of the campaign organics. It sounds obvious, but many marketers don’t do that.

Tip: Look back over the last few days/weeks/months and try to predict how many organics were going to come during the campaign. Don’t just take the average from the past: There might be seasonality, baseline growth…. If you see a trend, apply it.

2. It Mixes the Concepts of Organic and Trackable

It’s not just the baseline that needs to be accounted for. Many other channels are bringing customers, both organic and inorganic. Some are not caused by PAC, yet are inappropriately counted as such.

This begets the question: What is an organic user and what is an inorganic user?

What are Organic Users vs. Inorganic Users?

Think of cells: You can’t grow a full hair or a full nail from scratch. It grows little by little, cell division by cell division. That’s organic. Like an organism.

The business, like a body, grows organically when it develops itself. When a business executive says that a business grows “organically”, it means it’s growing by improving its own sales team, its own marketing campaigns, its own product features…

Conversely, a human adding a prosthetic leg, wearing plastic nails, using hair extensions, a wig made of human hair… This is not the body growing organically. It’s growing by purposefully adding something external, independent from the body’s current workings and size.

Similarly, a business growing inorganically is one that might, for example, buy other companies and absorb them.

It’s the same for online marketing: Organic channels are those that depend on the current state of the company. For example, viral acquisition depends on how many users you already have.

The same is true for word of mouth: Growth depends on how big the company already is in people’s mind.

SEO (search engine optimization, e.g., appearing on Google’s search results) depends on how many pages you have, how much you improve their performance for Google, and how much current users love them and interact with them, so SEO growth is organic too.

Conversely, display advertising, such as Facebook ads, retargeting, or SEM (search engine marketing, e.g., advertising on Google as opposed to appearing on its search results), is purely inorganic. The users you get this way are independent of the users you already have.

Certain channels stand in between. For example, on social media, if you pay for ads, that’s inorganic. However, if you build your own Instagram account, and you build your audience post by post.… Is that organic or inorganic? Your product is not really your Instagram account, but you’ve built it little by little. The bigger the audience, the faster it grows, so arguably it’s an organic channel.

The same goes for your email: You’ve built your email list over the years and now you have a body of addresses you can use to sell your products. But if you buy an email list — a list of email addresses to which you have a right to send emails — that’s inorganic.

What about Content Marketing? Arguably, you build your articles gradually over the years, gathering your audience for that content little by little. This is different from your product’s audience, but it’s an audience you organically grew nevertheless, so it’s an organic channel. However, if you pay influencers to create blog posts or Instagram ads praising your product, that’s inorganic.

Now, back to our attribution issue. Our goal here is to know how many users are coming because of our ads, even if they don’t come directly through ads.

You have some organic users that you can trackback to your organic sources, like email or virals or your social media account. You need to extract those trackable organics from the other ones, the ones that, really, you can’t track.

What you have left is untrackable organics. Well, not really. You shouldn’t call these remaining users untrackable organics, because in truth you don’t know whether they are organics or not. Just call them untrackable users. We don’t know where they’re coming from, or whether ads are playing an indirect role.

Tip: Make a list of all your channels, discuss with your team how to classify them, apply tracking to all of them if you don’t have it, and take them into account appropriately across all your analyses.

3. It Assumes Inorganic Users Cause All Untrackable Users

We’re done!

Most marketers stop here if they do anything at all. They extract the baseline and trackable users (both organic and inorganic). They then assume that the entirety of the new, untrackable users can be allocated to the ad spend.

Don’t make that mistake. As the change from “organic” to “untrackable” suggests, we just don’t know whether these people organically joined because of our ads or because of something else we couldn’t track. It’s wrong to assume it’s all due to the ads.

Let’s assume we had a crystal ball and could perfectly label all groups of users from where they come from:

With great tracking, you can have all ads properly tracked, as well as your SEO, your virals, your social media and content marketing.

But you can’t track word of mouth. You can’t track PR. App stores don’t always pass information about where a user really came from. So all the blue channels on the graph are untrackable. Unknowable. But clearly, you can see that some of them are caused by the ads, and some aren’t.

For example, PR was not caused by your ads. You just had a great Go to Market plan and received some great coverage that added some users. You also have a good relationship with Apple and Google and they gave you a featuring on their app store that meant a bunch of users. Those might be untrackable, but none of them was caused by the ads.

Conversely, word of mouth grew during the campaign. At least a part of that should be attributable to the ad campaign. Also, thanks to the ads, the company made it to the top of the app store charts, gaining a lot of users.

You have some trackable organics that can be attributed to ads: for example, SEO traffic went up. People might be seeing the ad and googling the company to learn more about it before buying. Conversely, a trackable organic like social media should not be attributed to ads.

Bottom line: You can’t assume that either organics or untracked users are caused by the ads.

4. It Doesn’t Account for the Impact After the Campaign

I said before that we had to take out the baseline from the lift caused by ads. Otherwise, you’re overweighting the impact of ads during the campaign.

But the opposite is also true: You should account for the lingering effect of ads.

For example, if your paid advertising campaign got a ton of users, and those pushed your mobile app up on the App Store, which promoted your product to many more users.… You might still be on the App Store charts for some time and get users from that. And all these new users will be talking about your product in the future to their friends if they like the app, even after you stop spending on ads. You need to attribute these new users to the ad campaign, both the ones from the App Store charts and those from word of mouth.

It’s not as straightforward as the drawing suggests, though: As discussed before, some of these users are trackable and some aren’t.

First, obviously, you extract the baseline, the users you received every day before the campaign.

You also extract trackable sources that you know aren’t caused by the ads (e.g., users from content marketing are likely not coming from your ads but rather from a video that you might have posted). Leave the trackable sources that you know are caused by your ads, such as a growth in your SEO traffic that can’t be explained for another reason.

You still have all sorts of untracked users, some of whom are due to your ad spend (like word of mouth), and some who aren’t (like PR). The way you solve that is that you should not calculate your organic ratio when you have other substantial untrackable inorganic sources of users. For example, if Apple features you on the App Store while you’re spending on ads, wait for another time to calculate your ratio of free users.

Now that you’ve extracted the baseline and the tracked users who don’t belong, and you’ve made sure there are no other untracked users, you’re ready to attribute the proper users back into the ad spend campaign, even after it’s finished. Just look at the curve and how it tapers out over the days until it goes back to your baseline. Attribute back all the additional users to your ad-spend campaign.

Tip: You should do that several times. You’ll see that your ratio is going to change every time, but it will hover around a number. That’s what you should pick.

5. You Can’t Model Organics After You Stop Your PAC Campaign

So far, we’ve talked about the organic ratio as a tool for ads attribution only: how to know which new users are caused by your spend. But this is not the only use of the organic ratio. People also use it for forecasting.

As a marketer, you want to be able to predict how many users you get in your product on any day — not just the days you are spending on ads.

But if your way of predicting new users is with the organic ratio, you’re at a loss. Since you’re multiplying every day the paid users with the organic ratio in order to calculate free users, what do you do when your paid users are zero? You can’t predict free users those days: You’re multiplying the ratio to zero. As a result, the organic ratio is worthless as a forecasting tool when a campaign is not running.

6. It’s Too Volatile

Even when a campaign is running, it’s not such a great metric. The graph below shows a true organic ratio for two similar products that were being advertised at the same time, for four months.

As you can see, for Product 3, the organic ratio hovered between 5 and 17. For Product 4, it was between 1 and 16. Put another way, you didn’t know whether each paid user was getting you one other user for free, or 16! Talk about a massive difference.

Not only that, but these two products were very similar and were using the same ad spend channels. Similar products, similar marketing, vastly different organic ratios. How are you supposed to use something that changes so much every day?

BOSS: “What’s your best guess on our ROAS (return on ad spend) next week, Jimmy?” JIMMY: “Well, boss, we spend $20 per new user and we get $2 from each, so it can be anything between losing 80% or earning 70%.” BOSS: “So, Jimmy, you’re telling me we can either lose a lot of money or make a lot of money?” JIMMY: “Yes, boss.” BOSS: “That’s helpful. Thank you.”

The day after, Jimmy started responding to LinkedIn requests from recruiters.

7. It’s Very Hard to Forecast Organic Users Well

If you can’t model user acquisition outside of ad spend — and even when you can, it’s extremely volatile—what’s the use of the organic ratio?

When a company grows, it needs to be able to predict how many users it acquires. The finance department becomes bigger and needs to be more and more accurate at predicting revenue and costs. The entire business depends on that.

I used to work in a company that spent a lot of money on ads. The team in charge of forecasting user acquisition was the digital marketing team. Obviously, they saw the world through the lens of paid acquisition, so they used their ad spend plus an organic ratio to predict user acquisition. They could do it because nearly every day we spent money on ads. That blinded them from the nonsense of doing it this way. But it was clear that their predictions were not very accurate.

This graph shows the predicted organic installs that we would get based on ad spend plus an organic ratio, and how it compares with reality. As you can see, the predictions didn’t match reality at all.

8. It Models Reality Very Poorly

The worst part is that it doesn’t match reality at all.

Do you think some people sign up for your product as soon as their friend does? It doesn’t happen like that.

Maybe a friend is using an app and raves about it at dinner. Or you oversee somebody playing with a game on the bus. Or you saw an ad that made you google a company and then you thought about it for a couple of days until you talked with your partner about it and decided to buy.

The word of mouth about a product happens with a delay from the download or the first time a user interacts with a product. The delay might be minutes, days, months… And it can depend on many factors: usage, a milestone reached some payback…

This is how things work in the real world, not how an organic ratio portrays it(1).

How to Properly Attribute Users to Ad Spend?

If using an organic ratio is not how you’re supposed to calculate the ROI of an ad spend campaign, how are you supposed to do it?

If you really want to do this well, you need a different approach.

1. The Bottoms-up Approach to Ad Spend Attribution

First, model each channel separately.

For channels where you can track all new users, that’s easier. You know every day, maybe even every hour, how many users are coming from that channel. Have a model to understand what drives these users. Project that to the future and see how your projection matches reality once you add the ad campaign.

Some of these tracked channels will be impacted by ad spend. For example, SEO. You need to model how that impact works with all the factors we’ve discussed before: e.g., extract a baseline before the campaign and add back users above the baseline even after the ad campaign is done. You can use a ratio for that. For example, through the ad campaign and after it’s done, I got 200 SEO users that I would not have received otherwise. Those are attributed to ads.

Once you have solved tracked users, prepare a test for untrackable. For example, one day when there’s no other event, keep everything the same and make an ad spend push, measuring how untrackable behave.

Beware: Don’t calculate a ratio during a PR campaign, during a featuring by Apple on the App Store, or while any other extraordinary untrackable event is in full swing. It will destroy your ratio.

Unfortunately, this approach is obviously laborious. Extracting all channels, measuring them separately, modelling them, taking into account everything that’s happening in the world to influence your untracked users…. All of that is not easy.

If only there was a better way…

2. The AB-Testing Approach to Ad Spend Attribution

If you really want to know the lift from your ad spend campaign, the golden standard will be AB-testing it.

If you keep a part of an audience without any ads, while another one sees your campaign, the difference in resulting user acquisition will be the true lift.

The hurdle is containing word of mouth. You can’t just randomly assign users to one bucket or another: If User A sees the ad and User B doesn’t, but user C comes to your site, and you can’t track back C to A because they talked about your product at brunch, how can you attribute C to the ad campaign that A saw?

One of the best solutions is geographic targeting: You take a few geographic areas and you open your ad campaign there. You measure the lift of the campaign in those areas specifically vs. other areas. If the lift you get from your ad spend campaign is consistent across all those geographic areas, you will know that is close to the true lift of your campaign.

Takeaways

Most companies measure the lift of their ad campaigns through an organic ratio, but there are many things wrong with that, such as the fact that it doesn’t account for a baseline, it mixes organic with untracked, it doesn’t account for users acquired after the campaign, and many other problems.

There are two ways to solve this:

1. Break down all channels, account for their traffic before and after the campaign, and model each one of them separately, allocating to the ad spend only the lifts that start at the same time and can’t be explained any other way.
2. You AB-test your campaign, ideally geographically.

Good luck with your attribution!

Tomas Pueyo is the VP of Growth at Course Hero, an Education Technology Unicorn.

(1) Word of Mouth is something you can try to model. It will depend on how the true word of mouth works for your product. I’ve successfully done it for video games, and I’ll write a post about that in the future

Note: This article focused specifically on the attribution of untracked users. For the proper attribution of users tracked through several channels instead (i.e., multi-channel attribution), geo-AB-testing is also the best approach. For a more thorough explanation, visit Avinash Kaushik’s article on the topic.