Incrementality: measure the lift attribution cannot prove
Use a worked holdout example to distinguish attributed sales from incremental lift, define a test, and interpret the result before changing spend.
By Mika Garcia · Published
In this article
Attribution assigns credit; incrementality asks what changed
Incrementality is the additional outcome caused by an intervention, compared with what would have happened without it. For advertising, that might mean additional purchases, qualified leads or contribution after costs. The difficult part is estimating the outcome you cannot observe for the same person at the same time: what they would have done without the campaign.
A customer can click an ad and then buy even if they already intended to purchase. An attribution model can correctly connect that payment to the click without proving the ad created the sale. This distinction matters when a campaign reaches existing customers or people already close to checkout.
Use attribution to inspect recorded journeys and reconcile revenue. Use a well-designed experiment to investigate causal impact. They answer complementary questions; a change in attribution rules is not an incrementality test.
Calculate lift with an explicit comparison
Consider a hypothetical randomized test with 10,000 eligible people in each group. The treatment group can receive the campaign; the control group is held out. After the same observation period, treatment records 500 purchasers and control records 400.
The observed rates are 5% and 4%. Their difference is one percentage point. Applied to the treatment population, the point estimate is 100 additional purchasers: 10,000 × (0.05 − 0.04). Relative lift is 25%: (5% − 4%) ÷ 4%. One percentage point and 25% describe the same result with different denominators.
These numbers illustrate arithmetic, not a real customer result. They do not establish statistical certainty. Assignment quality, spillover, sample size and uncertainty still determine whether the difference supports a decision.
Same-sized groups, different outcomes
Treatment
500 / 10,000 = 5%
Eligible to receive the campaign.
Control
400 / 10,000 = 4%
Held out from that campaign.
Normalize unequal groups before comparing
Do not subtract raw conversion counts when the groups differ in size. If treatment contains 20,000 people and control contains 10,000, compare their rates and apply the control rate to the treatment size to estimate its counterfactual baseline.
Also distinguish purchasers from orders. One person buying three times is one purchaser and three orders. Choose the outcome before looking at results, and keep returns, currencies, reporting windows and eligibility consistent. A changing definition can manufacture apparent lift.
Revenue needs its own measurement. Multiplying incremental purchasers by an average order value can be an illustration, but it is not a substitute for measuring revenue differences when order values vary.
Write a test brief before changing delivery
Start with one decision: whether a defined campaign adds enough value to justify its cost at the tested spend. Specify who is eligible, how assignment happens, the primary outcome, the observation period and how uncertainty will be reported.
Record other changes that could affect the result, including pricing, promotions, stock availability and channel budgets. Plan for conversion lag and interference between groups. A before-and-after sales chart alone cannot separate a campaign effect from seasonality or a simultaneous promotion.
- Define the intervention and the business decision it informs.
- Choose the unit of assignment and a credible control.
- Estimate the sample and duration needed for a meaningful effect.
- Freeze outcome definitions and analysis rules before results arrive.
- Record exclusions, delivery problems and deviations from the plan.
Three checks before calling a result actionable
1. Design
Comparable groups
Randomization or another defensible identification strategy.
2. Measurement
Same outcome definition
Equal windows, reliable outcomes and documented exclusions.
3. Decision
Effect plus uncertainty
Compare plausible lift and costs, not just a positive point estimate.
Translate the estimate into the decision you actually face
Suppose the illustrative test cost $5,000 in additional media and estimated 100 incremental purchasers. Incremental cost per purchaser is $50. That differs from dividing spend by all 500 observed treatment purchasers, which would produce $10 and answer another question.
An acquisition decision also needs margin, refunds and retention. A $50 incremental acquisition cost cannot be judged from a revenue-only return without knowing the economic basis. If the uncertainty range includes effects too small to cover costs, the next step may be better measurement rather than an aggressive budget increase.
Do not extrapolate a result at one spend level to every future budget. Audience saturation, creative, competition and season can change the response. Record the setting in which the estimate was obtained.
Use clean attribution records to support the investigation
Before a lift study, reconcile successful payments, refunds and timestamps against the source system. Check that the outcome is measured consistently across groups. Missing transactions can make a sound experimental design uninformative.
DATALYR can help inspect the campaign-to-payment records described in its attribution documentation. That is a measurement foundation, not a claim that its attribution reports create randomized holdouts or prove causal lift. Use an appropriate experimental system and analysis for the causal question.
Finish with a decision note: what changed, estimated effect and uncertainty, cost basis, limitations, and the next action. Keep the attributed result alongside the experimental result without treating either as a replacement for the other.