A/B Test Significance Calculator

Check whether your A/B test result is statistically significant: conversion rates, confidence intervals, z-score, p-value, and a plain verdict. Runs in your browser.

Your stuff stays yours. The conversion happens in this page's JavaScript. Nothing you paste here leaves your machine, and the tool keeps working with the network unplugged.

Paste your control on the first line and each variant below it - name, visitors, conversions - and get the analysis marketers usually pay a tool for: conversion rates with 95% confidence intervals, relative lift, z-score, two-tailed p-value, and a verdict in plain words. The math is the standard two-proportion pooled z-test, computed in your browser, so unreleased experiment numbers stay on your machine. More than two lines work too: each variant is tested against the control, which is the honest way to read a multi-arm test (testing every arm against every other arm inflates false positives).

How to analyze a test

In Variants in (name, visitors, conversions), put the control on the first line and one variant per line below it. Separate the three values with commas or tabs, so you can paste three columns straight from a spreadsheet.
Type visitor and conversion counts as plain whole numbers with no thousands separators (see the first problem below for why).
Press Analyze.
Read the Analysis out box. The CONVERSION RATES block lists each line's rate, raw counts, and 95% confidence interval. Below it is one block per variant compared against the control, with relative lift, z-score, two-tailed p-value, and verdict.

What the numbers mean and how they are computed

Each visitor is treated as one trial that either converts or does not. The conversion rate is conversions divided by visitors.

The 95% CI next to each rate is a normal-approximation (Wald) interval: the rate plus or minus 1.96 × √(p(1 − p) / n), clipped to stay between 0% and 100%. It works well with hundreds of conversions and poorly with very few. With zero conversions it collapses to 0.00% - 0.00%, which does not mean the true rate is known to be zero.

Relative lift is the variant's rate minus the control's, divided by the control's. It is relative, so moving from 5% to 6% is a lift of +20%, not +1%.

The z-score comes from a pooled two-proportion test. The calculator combines both groups into a single rate, p = (c₁ + c₂) / (n₁ + n₂), computes the standard error √(p(1 − p)(1/n₁ + 1/n₂)), and divides the difference in rates by it. The p-value is 2 × (1 − Φ(|z|)), where Φ is the standard normal distribution, calculated with the Abramowitz and Stegun approximation (accurate to about 1.5 × 10⁻⁷). The verdict says SIGNIFICANT when p is below 0.05.

One consequence surprises people: the two individual confidence intervals can overlap while the difference is still significant. Checking for overlap is a stricter test than the z-test, so read the p-value rather than eyeballing the intervals.

Worked examples

The placeholder data, A, 12000, 320 and B, 11800, 401, gives:

CONVERSION RATES
  A  2.67%  (320/12000)   95% CI 2.38% - 2.95%
  B  3.40%  (401/11800)   95% CI 3.07% - 3.73%

B vs A
  relative lift   +27.44%
  z-score         3.293
  p-value         0.0010   (two-tailed)
  verdict         SIGNIFICANT at 95% confidence

Now a closer case, pasted from a spreadsheet as tab-separated columns: Control with 5000 visitors and 250 conversions, Variant with 5000 and 300.

CONVERSION RATES
  Control  5.00%  (250/5000)   95% CI 4.40% - 5.60%
  Variant  6.00%  (300/5000)   95% CI 5.34% - 6.66%

Variant vs Control
  relative lift   +20.00%
  z-score         2.193
  p-value         0.0283   (two-tailed)
  verdict         SIGNIFICANT at 95% confidence

The intervals overlap between 5.34% and 5.60%, yet p = 0.0283 is below 0.05. This is the overlap effect described above.

With three arms, each variant is compared to the control separately and no correction is applied for making several comparisons. If you run many arms, a simple Bonferroni correction divides 0.05 by the number of comparisons. With four variants, treat only p-values below 0.0125 as significant.

Common problems and fixes

Thousands separators give wrong results

Commas separate the fields, so A, 25,010, 400 is read as name A, 25 visitors, and 10 conversions, and the extra value is ignored. Sometimes this produces the misleading error "conversions cannot exceed visitors", and sometimes it runs and prints a confident verdict about the wrong numbers. Always type 25010, and check that the counts in parentheses on the rates lines match your data.

A variant name gets cut off

A name containing a comma, such as Red, large button, splits in the same way. Use names without commas, such as red-large-button.

Conversions are counted per event, not per visitor

The test assumes each visitor either converts or does not. If you enter total orders or clicks, one visitor can count several times. The rate gets inflated, the variance is understated, and p-values come out too small. Count unique converting visitors, or users, consistently across all arms.

Units differ between arms

Mixing sessions for one arm and users for another, or pulling arms from different date ranges, breaks the comparison before any statistics run. Export every line from the same report with the same filters.

"Give at least two lines"

A single line has nothing to compare against. Add the control as the first line, then at least one variant.

Related tasks

Tag each variant's traffic source consistently with the UTM link builder, and check how shared test pages will look with the Open Graph preview.

Questions this tool gets asked

What test does this run?

A two-proportion z-test with a pooled standard error, two-tailed - the standard test for comparing conversion rates between two groups, and the same one most commercial A/B calculators run under the hood.

What does the p-value actually mean?

The probability of seeing a difference at least this large if the variants truly performed identically. Below 0.05, convention calls the result significant at 95% confidence. It is not the probability that B beats A - no frequentist test gives you that.

Why does my winning variant say “not significant”?

Small samples produce noisy rates. A +10% lift on 200 visitors per arm is well within chance; the confidence intervals shown will visibly overlap. The fix is more traffic per variant, decided before the test - not peeking until significance appears, which invalidates the p-value.

Can I compare more than two variants?

Yes - add more lines. Each variant is compared against the control (first line). Remember that testing many variants multiplies your chances of a fluke significant result; with 5+ arms, consider a stricter threshold like 0.01.