A/B Test Calculator — Significance & Sample Size
Two jobs on one page, because they are the same test at two points in time. “Test results” takes the visitors and conversions each variant got and runs a two-proportion z-test: conversion rates, absolute and relative lift, a p-value, a confidence interval on the difference, and a plain-English verdict. “Sample size” goes the other way — a baseline rate, the smallest effect worth detecting, your confidence level and power, and it tells you how many visitors each variant needs before you start.
Already ran the test? Enter what each variant got and find out whether the difference is real.
Test results
A — Control
B — Variant
Result
Enter visitors and conversions for both variants to see the conversion rates, the lift, and the p-value.
Turn the winning variant into a real design
Take the version that won and build the next round of creative on a Moda canvas.
Try Moda free →What the p-value actually means (and what it does not)
The p-value answers exactly one question: if the two variants were truly identical, how often would random assignment alone produce a gap at least this large? A p-value of 0.03 means “about 3% of the time.” That is all. It is not the probability that B is better than A, it is not the probability that you are wrong, and 1 − p is not the chance the variant wins. Those readings are the three most common misinterpretations in conversion optimisation, and they all inflate confidence in the same direction. The honest sentence is the awkward one: assuming no real difference exists, data this lopsided would be uncommon — so a real difference is a reasonable explanation.
Why peeking breaks the maths
A p-value is calibrated for one look at one sample size fixed in advance. Every extra look is another chance for random noise to cross the threshold, and noise crosses it often: run two identical variants against each other and check the result daily for a month, and you will “find” a significant winner surprisingly often, on data where the true difference is exactly zero. Stopping the moment a test goes green systematically ships wins that do not exist, which is why so many lifts fail to reproduce in revenue. Decide the sample size first, using the second tab, and read the result once. If you genuinely need to monitor a test continuously, you need sequential testing or a Bayesian approach — neither of which is what this calculator does.
How the numbers are calculated
Significance uses a two-proportion z-test with a pooled standard error: the two conversion rates are combined into one pooled rate under the null hypothesis, the standard error of the difference is sqrt(p(1−p)(1/n₁ + 1/n₂)), and z is the observed difference divided by that. The p-value is the normal tail beyond |z|, doubled for a two-tailed test. The confidence interval uses the unpooled standard error, which is the conventional pairing. Sample size uses the standard normal-approximation formula, n = (z₁₋α/₂·√(2p̄(1−p̄)) + z_power·√(p₁(1−p₁)+p₂(1−p₂)))² / (p₂−p₁)², with no continuity correction — expect a few percent difference against calculators that apply one. The normal CDF and its inverse are implemented from published approximations (Zelen & Severo for the CDF, Acklam for the quantile) and checked in tests against published reference values, so 95% two-tailed really is z = 1.95996, not 1.96 rounded off.
The assumptions you are making by using this
Four of them, and all four are routinely violated in practice. One: the metric is binary — a visitor either converted or did not. Revenue per visitor, session length, and order value are continuous and need a different test. Two: visitors were assigned randomly and independently, which breaks if the same person lands in both variants across devices, or if you split by day instead of by visitor. Three: the sample size was fixed before you looked (see above). Four: you are comparing exactly two variants on one metric. Testing four variants, or one variant on six metrics, multiplies the chances of a false positive and needs a multiple-comparison correction this page does not apply. This tool does the arithmetic honestly; it cannot tell you whether the experiment was sound, and it is not a substitute for statistical advice on a decision that matters.
Frequently asked questions
How many visitors does an A/B test need?
It depends almost entirely on two things: your baseline conversion rate and the smallest lift worth detecting. Lower baselines and smaller effects both need dramatically more traffic — detecting a 20% relative lift on a 5% baseline at 95% confidence and 80% power takes roughly 8,000 visitors per variant, while detecting a 5% relative lift on the same baseline takes well over 100,000. Use the sample-size tab with your real numbers before committing to a test; if the answer is more traffic than you get in a quarter, the honest move is to test a bigger change rather than to run the small one anyway.
What does 95% confidence mean here?
It is the threshold you are holding the result to: a p-value below 0.05 clears it. It does not mean there is a 95% chance the variant is better. It means that if the variants were identical, results this extreme would show up less than 5% of the time — so you are accepting a roughly 1-in-20 false-positive rate on tests where nothing is actually happening.
Should I use a one-tailed or a two-tailed test?
Two-tailed, in almost every case, and it is the default here. A two-tailed test asks whether the variants differ at all; a one-tailed test asks only whether B beats A, and buys its easier threshold by refusing to detect the opposite result. That is only legitimate if you committed to the direction before seeing any data and genuinely do not care about a loss. Switching to one-tailed after the numbers come in is p-hacking with extra steps.
My test is not significant. Should I keep it running?
Only if you decided in advance to run it that long, or if you are extending it to a new pre-committed sample size and will read it once more at that point. Extending a test repeatedly until it turns green is the peeking problem, and it manufactures winners out of noise. A non-significant result is a real result: it usually means the change was too small to detect with the traffic you have.
What happens if one variant got zero conversions?
The test still runs — a 0% arm is valid data. The relative lift is reported as undefined when the control is the arm that converted nobody, because dividing by a zero rate has no meaning; the absolute lift, the z-score, and the p-value are all still calculated. If both arms converted at exactly the same rate (including both at 0%), the p-value is 1: there is no difference to explain.
Does this work for revenue or average order value?
No. Both tabs assume a binary outcome — each visitor converted or did not. Revenue per visitor, order value, and time on page are continuous metrics with their own spread, and testing them with a proportion test gives a confidently wrong answer. Those need a t-test or a bootstrap on the underlying distribution.