Measurement & testing
Confidence interval
Also known as: CI
A range of values that, with a chosen level of certainty (usually 95%), contains the true effect size of an A/B test — a more informative metric than a single point estimate of lift.
A confidence interval is a range around a test's point-estimate lift that captures the uncertainty in the result. A "15% lift with 95% CI of [8%, 22%]" says the true underlying lift is very likely between 8% and 22%. CIs are far more informative than point estimates because they reveal the precision of the test — a tight CI means the result is well-sampled and reliable; a wide CI says the direction is promising but the magnitude is uncertain. Operators should always report CIs alongside lift numbers, especially when the CI crosses zero (which means the result is consistent with both "effect exists" and "no effect" depending on which end you land at). CIs narrow with larger samples, so for tests that land on marginal significance, the right response is "extend the test, widen the sample" rather than "declare inconclusive."