Measurement & testing
Incrementality testing
Also known as: holdout testing · control group testing
A measurement approach that holds out a randomly-selected group of eligible users from a lifecycle program and compares their behaviour to exposed users — reveals the true incremental impact of the program vs. organic baseline.
Incrementality testing is the only rigorous way to measure what a lifecycle program actually causes versus what would have happened anyway. Approach: randomly select a fraction of eligible users (often 5-10%) to be held out — they never receive the program. Exposed users receive the program as normal. The difference in outcome metric (revenue per user, retention rate, LTV) between exposed and holdout is the incremental impact. Without holdouts, "winback reactivation rate" mixes the program's causal contribution with the organic reactivation that would have happened from dormant-customer behaviour anyway — overstating program impact, sometimes by 2-5x. Holdout testing is the gold standard because it controls for confounders that A/B testing within the exposed population can't. Cost: the held-out users produce less revenue during the holdout window — a real trade-off for better measurement.
Read next
Holdout group design: the incrementality tool most lifecycle programs skip
Without a holdout, lifecycle ROI is attribution-model guesswork with a spreadsheet. With one, you get a defensible number you can actually put in front of finance. Here's how to size, run, and read a holdout — and the three mistakes that quietly invalidate the result.
Incrementality testing: the measurement that tells you if a program actually works
Last-click attribution makes lifecycle look bigger than it is. Incrementality testing strips out users who would have converted anyway and surfaces the real number. This is how to design a test that produces a figure you can defend in front of a CFO.