Measurement & testing
Holdout group
Also known as: global holdout · control group
A randomly-selected group of eligible users held out of a program — they never receive the lifecycle communications. Comparing their outcomes to exposed users reveals the program's true causal impact.
A holdout group is a randomly-selected share of the eligible audience that gets explicitly excluded from a lifecycle program. Unlike A/B test variants, holdouts are zero-exposure — they never get the emails, push notifications, or any communication the program produces. Purpose: measure the program's incremental impact versus the baseline behaviour of the same audience type. Typical holdout size: 5-10% of eligible users (smaller programs may run 15-20% for power; larger programs cap at 5% to minimise revenue drag). Holdouts are rotated quarterly in mature programs so individual users don't sit in holdout forever (which would itself be a retention risk). Holdout analysis produces the single cleanest number lifecycle teams can show leadership: "Our winback program drives $X incremental revenue per dormant user per quarter versus the no-send baseline."
Read next
Holdout group design: the incrementality tool most lifecycle programs skip
Without a holdout, lifecycle ROI is attribution-model guesswork with a spreadsheet. With one, you get a defensible number you can actually put in front of finance. Here's how to size, run, and read a holdout — and the three mistakes that quietly invalidate the result.
Incrementality testing: the measurement that tells you if a program actually works
Last-click attribution makes lifecycle look bigger than it is. Incrementality testing strips out users who would have converted anyway and surfaces the real number. This is how to design a test that produces a figure you can defend in front of a CFO.