An A/B test significance calculator tells you whether the difference between two conversion rates is likely to be real or just chance. Enter the visitors and conversions for each version and pick a confidence level. The tool runs a two-proportion z-test and gives you the p-value, the relative change and a confidence interval.
Loading the tool…
How to use the calculator
- Enter the number of visitors (or sessions, or ad clicks) that saw version A and how many converted.
- Do the same for version B.
- Choose a confidence level. 95% is the usual standard.
- Read the verdict. “Yes” means the difference is unlikely to be chance at that level; “Not yet” means you need more data or the versions perform about the same.
How the calculation works
The calculator compares the two conversion rates with a two-sided two-proportion z-test:
- It pools both groups to estimate a shared conversion rate.
- It works out the standard error of the difference and a z-score.
- It turns the z-score into a p-value: the chance of seeing a difference this big if the versions were really the same.
If the p-value is below 1 minus your confidence level (0.05 at 95%), the result is significant. The confidence interval shows the range the true difference probably falls in.
Mistakes that make A/B tests lie
- Stopping early. Checking every day and stopping the moment it’s significant inflates false positives. Decide the sample size first.
- Running too short. Run for at least one full week, ideally two, so weekday and weekend behaviour is included.
- Testing tiny changes on low traffic. Small uplifts need very large samples to detect.
- Changing the test midway. Edits to either version reset what you’re measuring.
- Ignoring the business impact. A significant 0.1% lift may not be worth the effort; a non-significant result with a wide interval may need a bigger test.
Example: a typical pattern when tests are checked every day. On day 3, with 400 visitors per version, B looks ahead with a p-value of 0.04. Stopping there would declare a winner. By day 14, with 3,000 visitors per version, the gap has closed and the p-value is 0.6. The early result was noise, which is why you set the sample size before you start.
Worked example
Version A: 5,000 visitors and 150 conversions (3.0%). Version B: 5,000 visitors and 190 conversions (3.8%).
The relative change is +26.7% and the p-value is about 0.027, below 0.05. So at 95% confidence, B is the winner. The 95% interval for the difference runs from about 0.09 to 1.51 percentage points, so the true lift could be small or quite large.
Frequently asked questions
What confidence level should I use?
95% is the common standard. Use 99% for expensive or hard-to-reverse decisions, and 90% only for quick, low-risk tests.
What is a p-value in A/B testing?
It’s the probability of seeing a difference at least this large if both versions actually converted at the same rate. A smaller p-value means the difference is less likely to be chance.
How many visitors do I need for an A/B test?
It depends on your current conversion rate and the smallest uplift you care about. As a rough guide, detecting a 20% relative lift on a 3% conversion rate needs over 10,000 visitors per version at 95% confidence and 80% power.
Can I use this for ad creative tests?
Yes. Use impressions or clicks as visitors and clicks or conversions as conversions, as long as both ads ran over the same period and audience.
Is this calculator free?
Yes. It runs in your browser and nothing you enter is stored.
Sources
Last updated: