Holdout group design: the incrementality tool most lifecycle programs skip

Attribution models — first-touch, last-touch, multi-touch — allocate credit across touchpoints that happened before a conversion. They do not answer the causal question: would the conversion have happened without any of those touchpoints? For lifecycle, the answer is often yes for a meaningful share of revenue. Users who would have returned, renewed, or bought anyway show up in attribution reports as lifecycle wins. Nobody's fault, exactly. Nobody's telling you either.

A holdout strips the confusion out at the root. Random assignment means the holdout cohort is, on average, identical to the messaged cohort except for message exposure. The revenue delta is incremental by construction. No attribution model required. No debate with finance about which model is "right".

5–10% of the eligible audience is the operator default, and for good reasons. 10% reaches statistical power faster. 5% loses less revenue if the program turns out to be a winner. For a program with tight monthly revenue targets, start at 5%. For a program where you're genuinely testing whether to keep running it, 10% gets you to the answer sooner.

The holdout has to be large enough to detect the effect size you care about. Ballpark: to detect a 10% incremental lift at 95% confidence over a quarter, you typically need around 5,000 users in the holdout — the exact number varies with baseline conversion rate. Below that the test is underpowered and you'll land on a number that isn't reliably different from zero, which is worse than running no test at all because people will still quote it.

The Orbit Experiment Design skill handles the power calculation against your specific baseline and expected lift. Skip it and you're running a test that can't answer the question it was built to ask.

Three rules for assignment matter more than most teams realise, because all three can break silently and you won't find out until someone asks a hard question in a QBR.

Assignment must be stable. A user assigned to the holdout today stays in the holdout for the full measurement period. Users flickering in and out contaminate the read. Use a persistent random integer — Braze's Random Bucket Number is built for exactly this — not a recalculated random value that shuffles on every audience refresh.

Assignment must be random."Users who haven't engaged in the last 30 days" is not a random cut. It's a selection bias with a severity rating. The holdout has to be statistically equivalent to the treatment group on every dimension except message exposure. RBN or equivalent. No shortcuts.

Assignment must be global.Exempting specific programs from the holdout — "we won't hold out onboarding because it feels cruel" — compromises the measurement. Either hold out or don't. Global holdout means every marketing send respects the same exclusion. Transactional sends are exempt, obviously.

Mistake 1 — holdout leakage. Users in the holdout occasionally get mail because a broadcast ignored the flag. 2% leakage is enough to invalidate the measurement — you're no longer measuring messaged versus unmessaged, you're measuring heavily-messaged versus lightly-messaged, and the delta shrinks accordingly. Audit broadcasts monthly for holdout compliance. Every month. No exceptions.

Mistake 2 — seasonal confounds. A holdout that runs only in November — i.e., Black Friday — will show enormous incrementality because of the volume spike. The number won't generalise to the rest of the year, but it'll get quoted as if it does. Run holdouts across full quarters or full years so you average seasonal effects out.

Mistake 3 — reading before statistical power. A two-week holdout result is almost never significant. Leadership asks for an update, the team produces a number anyway, and that number gets cited as the incrementality forever. The fix is simple and politically hard: don't publish interim reads. Publish once at the end of the measurement period, with the full analysis attached.

A holdout produces a single most-important number: incremental revenue per user. Multiply by audience size for total program contribution. Divide by program cost for ROI. This is the number that goes in front of finance and replaces the attribution-model figures that always, eventually, get questioned.

The retention economics guide covers how to frame the incrementality number in a CFO conversation. A defensible quarterly holdout study is usually more persuasive than six quarters of attribution spreadsheets, because it answers the causal question rather than the correlational one. Run one annually at minimum. Programs that run holdouts annually have budget conversations that go differently from programs that don't.

On the mechanics question that comes up constantly — in Braze, use Random Bucket Number filters. A fixed slice (RBN < 500 for a 5% holdout) is stable, random, and respected across every program in the instance. The same attribute that underpins IP warm-up is what you want here. It was designed for this job.

What if the holdout shows zero incremental lift? Don't panic. Don't bury it either. Zero lift is honest information, and it's worth investigating before it turns into a narrative. Is the program targeting users who were going to convert anyway? Is the offer too weak to change behaviour? Is the timing wrong? Zero is rarely the final answer on a well-designed program, but when it is, the program needs rethinking — not another quarter of the same cadence with a redesigned header image.