Statistical Significance
Z-test for A/B/n experiment results. Compare a control against up to 9 variants.
This tool is part of Orbit. A free Claude Desktop extension with 50+ lifecycle marketing skills. Pay what you think it's worth once it saves you a day.
Control (A)
Variant B
About the Statistical Significance Calculator
Free A/B test statistical significance calculator using a two-proportion z-test. Supports up to 9 variants against 1 control, with confidence, lift, and p-value for each. Built for marketers running email, push, in-app, and landing page experiments.
How to use
- Enter your control sample size and conversion count.
- Enter up to 9 variants with their sample sizes and conversions.
- Choose a confidence level (90%, 95%, 99%).
- Read confidence, lift, and p-value for each variant.
Common use cases
- Email A/B/n tests (subject line, from name, content variants)
- Push notification experiments
- Landing page variant testing
- In-app message experiments
Who it's for
Lifecycle and growth marketers who run experiments and need a quick significance check before calling a test.
Frequently asked questions
- What is statistical significance in A/B testing?
- Statistical significance is the probability that an observed difference between variants is not due to random chance. Commonly expressed as a p-value; p < 0.05 means 95% confidence that the difference is real.
- What test does this calculator use?
- A two-proportion z-test, which is the standard for comparing conversion rates between two groups when sample sizes are large enough.
- How large does my sample need to be?
- The z-test assumes each group has at least ~30 conversions. Below that, confidence intervals widen and results become unreliable.
- What's the difference between confidence and p-value?
- They're complements: 95% confidence = p < 0.05. Both express the same idea from different angles.
- Can I run more than one variant?
- Yes. This calculator supports up to 9 variants vs 1 control. When comparing many variants, consider a Bonferroni correction to avoid false positives.
Related Orbit tools
Built into Orbit for Claude
Stop copying numbers into a calculator. Orbit does this end-to-end.
Inside Orbit for Claude, the Experiment Design skill runs significance testing natively. It pulls sample sizes and conversions straight from your Braze workspace, applies the right test, and flags when a result is or isn't ready to call. No spreadsheets, no copy-paste.