BYTETOOLS

A/B Test Significance Calculator

Test whether an A/B result is statistically significant with a two-proportion z-test: p-value, z-score, confidence interval and relative uplift. Free tool.

A — control

B — variant

5.000%
Control rate
6.200%
Variant rate
+24.00%
Relative uplift
+1.200 pp
Absolute difference
2.6096
z-score
0.0091
p-value
99.09%
Confidence
1.9600
Critical z at 95%

Statistically significant at 95%. A difference this large would turn up by chance about 0.0091 of the time if the two versions were really identical, which is below your 5% threshold.

GroupVisitorsConversionsRate
A — control5,0002505.000%
B — variant5,0003106.200%
Pooled10,0005605.600%

95% confidence interval on the difference

0.299 pp to 2.101 pp

The interval excludes zero, which agrees with the significance verdict above.

z = (p̂B − p̂A) ÷ √(p̂(1−p̂)(1/nA + 1/nB)), where p̂ is the pooled rate  ·  CI = (p̂B − p̂A) ± z* √(p̂A(1−p̂A)/nA + p̂B(1−p̂B)/nB)

Worked example: 100 conversions from 1,000 control visitors (10%) against 130 from 1,000 variant visitors (13%). The pooled rate is 230 ÷ 2,000 = 11.5%, the standard error is 0.0142671, so z = 0.03 ÷ 0.0142671 = 2.1027 and the two-tailed p-value is 0.0355 — significant at 95% but not at 99%.

This is the standard two-proportion z-test, the same maths most A/B testing dashboards use. The p-value comes from a Chebyshev approximation to the complementary error function that is accurate to seven significant figures, so very small p-values are shown as “< 0.0001”. Two caveats worth taking seriously: peeking at a running test and stopping the moment it crosses your threshold inflates false positives badly, and a significant result on a tiny absolute difference can still be commercially meaningless. Everything is computed in your browser and no test data is uploaded.

What is the A/B Test Significance Calculator?

Before you ship a winning variant, it is worth knowing whether it actually won. This calculator runs the standard two-proportion z-test on your control and variant numbers: it pools the two conversion rates, computes the z-score and turns it into a p-value, then tells you plainly whether the difference clears your chosen confidence level.

  • Two-proportion z-test with the pooled standard error, the industry standard
  • p-value, z-score, confidence percentage and the critical z side by side
  • Confidence interval on the difference in conversion rates
  • One-tailed and two-tailed options, and confidence levels from 80% to 99%
  • Summary table of control, variant and pooled rates
  • Runs entirely in your browser — your experiment data is never uploaded

How to use the A/B Test Significance Calculator

  1. 1

    Enter the visitor count and conversion count for your control in the A panel.

  2. 2

    Enter the same two numbers for your variant in the B panel.

  3. 3

    Pick a confidence level — 95% is the usual default.

  4. 4

    Choose two-tailed if either version could win, or one-tailed if you only care about B beating A.

  5. 5

    Read the significance verdict, then check the confidence interval to see the plausible range of effects.

About the A/B Test Significance Calculator

Before you ship a winning variant, it is worth knowing whether it actually won. This calculator runs the standard two-proportion z-test on your control and variant numbers: it pools the two conversion rates, computes the z-score and turns it into a p-value, then tells you plainly whether the difference clears your chosen confidence level.

Alongside the verdict you get the relative uplift, the absolute difference in percentage points, and a confidence interval on that difference so you can see the range of true effects your data is consistent with. One- and two-tailed options are both available, and the confidence level is a dropdown from 80% to 99%.

The p-value comes from a Chebyshev approximation to the complementary error function that stays accurate to seven significant figures deep into the tail, so even very small p-values can be trusted rather than rounding to zero. Everything is computed in your browser: no test data is uploaded, nothing is stored, and there is no account to create.

Frequently asked questions

How do you know if an A/B test is statistically significant?

Compare the p-value against your significance threshold. At 95% confidence the threshold is 0.05, so a p-value below that means a difference this large would rarely appear by chance if the two versions were identical. This calculator does the comparison and states the verdict directly.

What does the p-value actually mean?

It is the probability of seeing a difference at least this large purely by chance, assuming the two versions really perform the same. A p-value of 0.03 does not mean there is a 97% chance your variant is better — it means data this extreme would show up 3% of the time even with no real difference.

Should I use a one-tailed or two-tailed test?

Two-tailed is the safer default because it tests for a difference in either direction, which is what you actually want when a variant could plausibly perform worse. A one-tailed test is easier to pass and should only be used if you decided before running the test that you only care about improvement.

Why does my test say not significant when the variant is clearly winning?

Almost always sample size. A three-point difference on a thousand visitors per arm is well within normal random variation; the same difference across ten thousand per arm usually is not. Small samples need large effects to reach significance, which is exactly what the test is protecting you from.

Can I stop the test as soon as it becomes significant?

No — this is the single most common mistake in A/B testing. Repeatedly checking a running test and stopping the moment it crosses your threshold badly inflates false positives, because random fluctuation will cross the line eventually. Fix your sample size in advance and check once at the end.

Is statistical significance the same as a meaningful result?

No. With a large enough sample, a difference far too small to matter commercially can still be significant. Always look at the absolute difference and the confidence interval alongside the p-value before deciding whether a win is worth shipping.

Related tools