A/B Test Significance Calculator
Check whether your split test has a real winner, or plan how many visitors a new test needs before you start it.
Version A (control)
Version B (variant)
- Rate A
- 5.00%
- Rate B
- 7.00%
- Uplift (B vs A)
- +40.0%
- p-value
- 0.0597
Not significant yet, keep the test running. Confidence is 94.0%, below the 95% target.
Two-sided two-proportion z-test, z = 1.88. Significant means a p-value under 0.05 (95% confidence).
How the Calculator Works
- Conversion rate = conversions / visitors, for A and for B
- Uplift = (rate B - rate A) / rate A
- z = (rate B - rate A) / standard error, using the pooled rate of both versions
- p-value = the two-sided chance of a gap this big if A and B were really the same
- Sample size per variant uses the standard two-proportion formula at 95% confidence and 80% power
Worked example
A converts 50 of 1,000 visitors (5.0%) and B converts 70 of 1,000 (7.0%). That looks like a 40% uplift, but the p-value is 0.06, above 0.05, so it is not significant yet.
To reliably spot a 20% lift on a 5% conversion rate (5% to 6%), you need about 8,158 visitors per variant, or 16,316 in total. With 1,000 visitors a day, that is about 17 days.
A/B Testing Questions, Answered
What does statistical significance mean in an A/B test?
It tells you how surprising your result would be if A and B really performed the same. A p-value under 0.05 (95% confidence) means a gap this big would show up by chance less than 5% of the time. It does not prove B is better by exactly the uplift you see, and it says nothing about whether the gain is big enough to matter for your business.
Why shouldn't I stop a test as soon as it looks significant?
Results swing a lot early on. If you check every day and stop the first time the p-value dips under 0.05, you will declare far more false winners than 5%. Work out the sample size before you start, run the test until each variant reaches it, and ideally cover at least one full week so weekday and weekend visitors are both included.
How is the sample size worked out?
The planner uses the standard formula for comparing two conversion rates with a two-sided test at 95% confidence and 80% power. It needs your current conversion rate and the smallest relative lift you care about. Smaller lifts and lower baseline rates need many more visitors, because the difference is harder to tell apart from random noise.
Should I use a one-sided or two-sided test?
This calculator uses a two-sided test, which checks whether B is better or worse than A. A one-sided test only looks in one direction, so it reaches significance with fewer visitors, but it cannot warn you when the change hurts. Unless you decided on one direction before the test and would never act on a drop, two-sided is the safer choice.
How can I A/B test with low traffic?
Test bigger, bolder changes so the lift you are looking for is larger, since that cuts the sample size sharply. Measure an action closer to the change, such as clicks on a button, instead of final sales. Test on your busiest pages, run tests for longer, and avoid splitting traffic across more than two variants.
Want Tests That Find Real Winners?
TML plans, runs and reads A/B tests on your pages, so each change you keep is backed by data.