Incrementality Measurement
What
Incrementality is the lift in a desired outcome (ex. business revenue) that occurs as a result of marketing activities (ex. paid advertising). We run incrementality tests to measure the true impact of our marketing strategies.
Why
Without a formal incrementality test approach, many factors limit our ability to measure marketing accurately, including attribution bias, limited measurement of top of funnel and cross-channel activity (amplified by 3rd party cookie deprecation), the influence of confounding micro and macro factors, etc.
Incrementality tests reduce our reliance on siloed sources of truth, and help us redirect marketing strategy to meet business objectives.
Q: What data can we use to quantify the true impact of advertising?
Questions We Can Answer with Incrementality Tests
What impact will launching X channel have on business results?
Business ABC has never run programmatic ads and would like to understand the incremental revenue and ROAS from launching programmatic display for two months.
What is the impact of our current paid media on business results?
Business PQR is currently running a mix of Google search (Brand, NonBrand, Competitor), Youtube and Linkedin and would like to understand the incremental revenue and ROAS of their entire program by turning off or reducing spend. Business CDE is questioning the value of Meta ads and wants to scale or pause this channel, but do so in a measurable way.
What is the impact of paid media at an increased spend?
Business XYZ is currently running ads across Google Search, Youtube and Facebook and would like to understand the incremental revenue and ROAS by increasing spend across advertising by 50%
Is X media mix more impactful than Y media mix?
Business FGH is currently running acquisition efforts across Meta and Youtube in addition to Brand search and retargeting, and would like to understand which acquisition channel to scale based on each channels incremental sales and cost efficiency.
Scenario Example #1
“What impact will launching channel x have on business results?”
Business ABC has never run programmatic ads and would like to understand the effect to revenue and other channel traffic from launching programmatic display for two months.
Suggested Test Design
| Cell | Action |
|---|---|
| Holdout (X% geos) | Maintain business as usual advertising program |
| Test (X% geos) | Launch ad channel |
Top Consideration
Ensure you have enough budget planned for testing new channels. Thrive can advise on an appropriate estimated test budget and length to measure a meaningful effect. Concentrating test budget across smaller geo groups can improve effect size (ex. 80% geos holdout, 20% geos test vs 50-50%).
Scenario Example #2
“What is the impact of our current paid media on business results?”
Ex 1. Business PQR is currently running a mix of Google search (Brand, NonBrand, Comp), Youtube and Linkedin ads and would like to understand the incremental revenue and ROAS of their entire program by turning off or reducing spend. Ex 2. Business CDE is questioning the value of Meta ads and wants to de-fund or pause this channel entirely, but to scale down in a measurable way.
Suggested Test Design
| Cell | Action |
|---|---|
| Holdout (X% geos) | Maintain business as usual advertising program |
| Test (X% geos) | Pause ad spend |
Top Consideration
Factor in an expected loss or reduction in business results from pausing ads across select geos. Testing an ‘ad outage’ across a smaller proportion of geos can reduce your opportunity cost (ex. 80% holdout, 20% pause vs 50-50%).
Scenario Example #3
“What is the impact of paid media at an increased spend?”
Business XYZ is currently running ads across Google Search, Youtube and Facebook and would like to understand the incremental revenue and new user growth to be expected by increasing spend across advertising by 50%.
Suggested Test Design
| Cell | Action |
|---|---|
| Holdout (X% geos) | Maintain business as usual advertising program |
| Test (X% geos) | Scale ad spend |
Top Consideration
Plan to scale for strategic growth: broad acquisition audiences are good options for scaling as opposed to tactics that are already fully funded, or where we expect high diminishing returns and/or lower incremental value (ex brand search).
Scenario Example #4
“Is X media mix more effective than Y media mix?”
Business FGH is currently running acquisition efforts across Meta and Youtube ads in addition to Brand search and Retargeting, and would like to understand which acquisition channel to scale based on each channels’ incremental sales and cost efficiency.
Suggested Test Design
| Cell | Action |
|---|---|
| Holdout (X% geos) | Maintain business as usual advertising program |
| Test (X% geos) | Scale YT acquisition, maintain other channel spend |
| Test 3 (X% geos) | Scale FB acquisition, maintain other channel spend |
Top Consideration
Different ad channels can contribute to business success at different budgets, different moments in the user journey, and with differing latency, so apples to apples comparisons can be counterproductive. Optimize to the best ad channel mix as opposed to the ‘best’ ad channels.