A/B Test Significance Calculator

Check whether your test's result is real or just noise, with both a frequentist p-value and a Bayesian probability that B beats A.

Your numbers

A (control)

Visitors

visitors

Conversions (orders)

B (test)

Visitors

visitors

Conversions (orders)

Confidence level

Your result · updates as you type

Not yet significant

20% relative

A converts at 4%, B at 4.8%. Two-sided p-value is 0.051, above your 5% threshold.

P-value: 0.051
Bayesian: P(B beats A): 97.4%
95% CI of the difference: -0% to 1.6%

The p-value hasn't crossed your 5% threshold yet. Either run the test longer, or use the Bayesian probability as a softer read: 97% chance B beats A right now.

Breakdown

A: conversion rate: 4%
B: conversion rate: 4.8%
Absolute difference (B − A): +0.8%
Relative uplift (B vs A): +20%
Frequentist p-value (two-sided): 0.0511
Bayesian probability B beats A: 97.4%

Two ways to read the same test

The frequentist view asks: if A and B truly converted the same, how likely is a gap this big by chance alone? That's the p-value, from a two-proportion z-test. The Bayesian view asks directly: given what we've seen, what's the probability B actually converts better than A?

Example: the default numbers

A (control): 200 conversions from 5,000 visitors (4.0%)
B (test): 240 conversions from 5,000 visitors (4.8%)

B is 20% higher, relatively. The p-value is 0.051, just above the usual 5% threshold, so it's not yet significant at 95% confidence. The Bayesian view is softer: 97.4% probability that B genuinely beats A.

The two views often roughly agree, and here they point the same direction (B looks better) even though only one crosses its usual threshold. That's normal near a borderline result, and it's a sign to keep the test running rather than to trust either number too much yet.

Running the test fairly

Decide your sample size and confidence level before you start, with the sample size calculator. Checking the p-value every day and stopping the moment it looks good inflates your false-positive rate.
Check the traffic split first. If close to half the visitors didn't land in each variant, run the sample ratio mismatch checker before trusting any result here.
Run full business cycles. A week that includes a weekend and a payday looks different from one that doesn't. Most tests need at least one full week, often two.

Questions people ask

What does a p-value of 0.05 mean?

It means there's about a 5% chance of seeing a gap this large (or larger) between A and B if they actually converted identically. Below 0.05 is the common (not universal) cutoff for calling a result significant.

Why do the frequentist and Bayesian results sometimes disagree?

They answer different questions and use different assumptions. Near a borderline p-value, like 0.05, it's common for the Bayesian probability to look more confident than the frequentist test allows. Treat disagreement as a sign to keep testing, not a reason to pick whichever number you prefer.

Can I stop the test early if it looks significant?

Checking daily and stopping as soon as p < 0.05 is one of the most common ways to fool yourself in A/B testing: it inflates the real false-positive rate well above 5%. Decide your sample size upfront with the sample size calculator, and stick to it.

What confidence level should I use?

95% is the common default. Use 99% for changes that are expensive or risky to roll back, and 90% only when a false positive is cheap to correct.

Last reviewed 30 September 2026.