Run fewer incrementality tests and get more from them

Incrementality testing is having a moment. And for good reason. Platform attribution has obvious limitations, MMM can’t answer every tactical question, and marketers are increasingly realizing that just because a platform says it drove a sale doesn’t mean that sale wouldn’t have happened anyway.

But you need a systematic approach to ensure you’re testing the right things and acting on the results.

I use a framework for this called IDEATE. Despite my feelings about cute acronyms, I’ve used this one enough to see its value.

IDEATE stands for Insight, Draft Hypothesis, Envision Paths, Arrange the Test, Track Results, and Execute on Findings.

Most of those are self-explanatory. So let’s focus on the two where teams most often mess up testing: Insight and Envision Paths.

As incrementality testing becomes more common, there’s a temptation to test every channel and every nuance within them. Unfortunately, you can’t. Testing capacity is limited, so spend it on questions where uncertainty has meaningful financial consequences.

You have to be certain why you are considering this test in the first place. What have you seen in your data, MMM, observational analyses, or through operating the business that makes you question what you believe?

There is a place for instinct

If you’ve been around a marketing program for a while and something seems off about a channel or tactic, trust that instinct and investigate. That doesn’t mean your instinct is right. That’s why you’re testing it. But there is probably a reason something feels off.

Say that every time we increase spend on Meta ASC, Meta continues to report attractive marginal CPAs. But at the same time, our new customer CAC is getting worse. That should get our attention.

Maybe Meta is finding incremental customers efficiently, and something else is driving CAC higher. But maybe as we scale, Meta is finding customers who would have purchased anyway and taking credit for those conversions.

That is a question worth spending testing capacity on because understanding it can have real P&L impact. If we find ASC is highly incremental at the current spend level, great. We can keep spending and maybe scale further. If we find we’re out on the diminishing-returns curve, we can reduce spend and find a better use for those dollars.

By contrast, a channel supported by your MMM, observational analysis, and sales response doesn’t deserve the next testing slot. Use your limited testing capacity where there is both uncertainty and a material P&L consequence.

Decide what you’ll do before you know the answer

Once we have an insight, we draft a hypothesis and figure out how we might test it. But before arranging and running the test, IDEATE forces a step that is missing from many experimentation programs: Envision Paths.

Before you run the test, write down the possible results and what you’re going to do in each scenario. Not after the readout. Before the test starts.

This reduces bias. It’s easy to rationalize a result you don’t like after you see it, especially when it suggests cutting spend in a channel that a team, agency, or platform has spent years building.

A test only creates value if you do something with the result.

Say I’m running a test on a large paid social program. I have a 50% pre-advertising contribution margin, and I need roughly a 2.0 incremental ROAS to break even. Before I run the test, I might agree on something like this with the team:

• If iROAS comes back above 3.0, we’re going to increase spend 50%. 
• If it comes back between 2.5 and 3.0, we’ll increase 25%. 
• Between 2.0 and 2.5, we’ll maintain spend or scale slightly. 
• Between 1.5 and 2.0, we’ll reduce spend 25% and build an optimization plan. 
• If we see very little lift and iROAS is below 1.5, we’ll make one or two fundamental changes before putting meaningful dollars back into the existing approach.

The thresholds will vary by channel, spend level, and business. What matters is agreeing on them before you know the result.

Otherwise, here’s what happens: We run the test, and it comes back worse than we hoped. We spend 45 minutes discussing the methodology and why the result might not be perfect. We agree it was an “interesting learning.” We keep spending the same amount of money.  

Make the test useful after it ends

An incrementality test should do more than tell you whether to change spend when the results arrive. Use it to put your everyday performance metrics in context until you have better evidence.

Say Meta reports a ROAS of 4.0 during the test, while the incrementality test measures an iROAS of 2.0. At that spend level and under those conditions, Meta’s reported return is twice the incremental return measured by the test.

Don’t assume that relationship will hold forever. But you can use it as a working benchmark while the campaign and measurement approach remain similar.

Then platform ROAS falls to 3.5. If the relationship observed in the test still holds, iROAS may have fallen to around 1.75. With a 50% pre-advertising contribution margin, those incremental dollars may no longer pay for themselves.

Now you have a reason to act. Reduce spend, optimize the campaign at the new spend level, or make a larger change to the strategy and run another incrementality test.

The test becomes a reference point for managing the channel between tests. Rerun it when the campaign or measurement conditions change enough that the original relationship may no longer apply.

Finish the loop

The remaining IDEATE steps cover test design, documenting the results, and acting on what you learned. Keeping those findings in one place also gives future tests a starting point, rather than forcing teams to relearn the same lessons.

Start with a real insight, prioritize questions with meaningful financial consequences, and decide what you’ll do before you know the answer. You may run fewer incrementality tests, but each one has a better chance of changing the business.