Blogs / How Can I Test What a Marketing Channel Really Adds?

How Can I Test What a Marketing Channel Really Adds?

Nov 16, 20267 min read
Pulkit Khurana

Founder, SproutMe

A line drawing of a closed sluice gate holding back water, illustrating how a holdout test marketing experiment isolates what a channel adds.

Your retargeting campaign reports a return you'd happily show the board, yet you suspect most of those buyers were already on their way to checkout. Pausing it for a fortnight won't settle the question, because seasonality and promotions move sales as much as the ad does.

To test what a channel really adds, withhold it from a matched group of people or regions while everyone else sees it, then compare conversions. The gap between the two groups is the channel's incremental contribution, a number no attribution model can produce. You can run it through a platform's lift study or as a geo-lift test of your own.

What does a holdout test actually measure?

Attribution answers a bookkeeping question: which touch gets credit for a sale. A holdout answers a causal one: would the sale have happened without the ad? Most wasted budget sits in the gap between those two answers.

Take branded search. A shopper who searches your brand name and clicks the paid ad would often have clicked the organic listing directly below it. Attribution gives the ad full credit, but the business gained nothing. The same logic explains why every ad platform says it brought the sale. Each one counts touches, not causes.

A holdout test builds the missing counterfactual. You split an audience or a set of markets into two comparable groups, expose only one to the channel, and measure both over the same period. Both groups face the same season, the same promotions and the same competitor activity, so the only systematic difference left is the channel. Treatment conversions minus control conversions gives you incremental conversions: the sales that exist because the channel ran.

That comparison is the whole method. Every other decision, from how you split the groups to how long you run the test, exists to keep the two groups genuinely comparable.

Should you use a platform lift test?

The fastest route is the one the ad platform already offers. At the time of writing, Google's Conversion Lift documentation describes holding a control group of users back from your ads and comparing their conversions with a treatment group that saw them. Crucially, it ignores your standard conversion tracking settings and attribution rules. It counts every conversion across the full study, not only those inside a window such as seven-day click-through. Other platforms run similar managed studies, and some show the control group a neutral placeholder ad so both sides see the same volume of advertising.

Platform studies are cheap to start, they handle user-level randomisation for you, and they need no in-house statistician. They also carry a conflict you shouldn't ignore. The platform designs the test, runs it, and is the channel being graded. It can only test itself, so it will never tell you how its lift compares with a channel on someone else's network. Treat its result as a strong signal about one channel, not as a referee between channels.

How do you run a geo-lift test yourself?

Test by region when you want a result the platform doesn't control, or when the channel has no lift product, such as TV, out-of-home or a podcast sponsorship. A geo-lift test runs the channel in some markets and keeps comparable markets dark. Three steps carry most of the weight.

  1. Match the markets. Pair test and control regions on sales history, demographics and market size.
  2. Keep controls dark. Run the campaign only in test markets. Any spend that leaks into a control region shrinks the gap you're trying to measure.
  3. Compare after launch. The difference in performance between test and control over the test period is your incremental sales.

Before launch, plot four to eight weeks of pre-launch sales for both groups, a window Raphael Weishaupt's practical guide to geo-lift recommends. The two lines should move together. If they diverge before you've spent anything, the markets aren't matched, and no post-launch difference will mean much.

Run the test past your normal purchase cycle so delayed conversions land inside it. Across successive tests, rotate which markets serve as test and control. Otherwise a quirk of one region gets baked into every result.

Be honest about the cost. A geo test deliberately withholds media from real customers, so you give up whatever sales the channel would have driven in the control markets. It also needs enough volume in each market to separate signal from noise. That makes it a better fit for larger spenders and bigger channels. A brand with thin regional sales may not be able to read a result at all.

How do you read a lift result honestly?

Wait for the study to end. Cumulative numbers often appear mid-study and early reads swing, which is why Google recommends waiting until the close for the most accurate lift result.

Then turn lift into money. Divide spend by incremental conversions to get incremental cost per action. Google's documentation gives a hypothetical: a $10,000 campaign that produces 800 incremental conversions has an incremental CPA of $12.50. Set that beside the CPA your normal dashboard reports. The distance between the two shows how much credit attribution was giving the channel for sales it didn't cause.

Treat relative lift with care. It's calculated as incremental conversions divided by control conversions, and it can range from −1 to infinity. A study with a very low control count produces dramatic percentages from a handful of extra sales, and Google warns that relative lift is easily misread across studies. Compare channels on incremental cost, not on lift percentages.

Before the test starts, decide what you'll measure and which result would change your budget. It's easy to choose test and control groups, consciously or not, in ways that leave the result ambiguous. A test nobody committed to acting on is just an expensive report.

Which channel should you test first?

Start where the money is and where attribution is most likely to flatter. Your highest-spend channels carry the most budget at risk. Retargeting and branded search are the classic suspects because both reach people who already intend to buy.

Use the result to recalibrate your day-to-day reporting, not to replace it. Attribution stays the signal you watch each week, and the holdout tells you how far to discount it. If a channel's incremental CPA comes in well above its reported CPA, read its dashboard with that discount in mind and move budget accordingly.

Then repeat the test. Audiences, creative and competition shift, so last year's lift figure describes last year's channel. A regular cadence of tests on your major channels keeps the discount honest. Stitched together, those results answer which marketing channel is actually bringing you customers far better than any single model can.

Conclusion

Attribution tells you who touched the sale. Only a holdout tells you whether the channel caused it. A platform lift study is the quickest way in and a geo test is the most independent, and both depend on the same discipline: comparable groups, a full test window and a decision committed to in advance. Read the result as incremental cost, set it against what the dashboard claims, and move budget on the difference.

Once you know what each channel really adds, SproutMe Plan models how budget would perform across Google, Meta, TikTok and LinkedIn under different scenarios before anything is committed.

Frequently Asked Questions

Yes, if you can get sales data at the right level. A geo-lift test compares regions, so it can measure brick-and-mortar impact when retail partners share store-level or geo-level sell-through data. Match test and control markets on the same basis you would online, and run the campaign only in the test regions. Without regional sales data, the comparison can't be made.

Some purchases happen after the study window closes. For Demand Gen-only studies, Google's Conversion Lift adds "delayed incremental conversions" by default. These are modelled estimates of conversions expected after the study's official end date, added to the report on its last day. Treat that portion as an estimate rather than an observation, and run your own tests past your normal purchase cycle.

Start with orders. Conversion value varies widely from one purchase to the next, so incremental conversion value is more volatile than incremental conversions and produces larger confidence intervals, as Google notes for its own lift studies. A handful of large orders can make a revenue lift look significant or flat. Switch to value once conversion counts give you a stable read.

Grow smarter with AI marketing tips

Join our newsletter to get practical insights, automation ideas, and performance tips straight to your inbox.

Get a complimentary audit to uncover AI opportunities hidden in your data.

Put these strategies to work