A/B test sample size calculator

How many visitors does an A/B test need to give a result you can trust, and how many weeks will it run? Enter your current rate and the smallest lift you want to detect.

The numbers you enter stay in your browser; they are not sent or stored anywhere.

Result

Visitors per variant
21,109
Total visitors
42,218
Rate to detect
2.4%
Duration (full weeks)
3

At 95% confidence (two-sided) and 80% power. Run the test for 3 full weeks and do not read the result before you reach this number.

Formula

  • p₁ = current rate · p₂ = p₁ × (1 + relative lift) · p̄ = (p₁ + p₂) ÷ 2
  • n = [1.96 × √(2 × p̄ × (1 − p̄)) + 0.84 × √(p₁ × (1 − p₁) + p₂ × (1 − p₂))]² ÷ (p₂ − p₁)²
  • duration = 2 × n ÷ weekly visitors, rounded up to a full week

The standard sample size formula for comparing two proportions. Confidence is fixed at 95% (two-sided, z = 1.96) and power at 80% (z = 0.84). n is the number of visitors each variant needs; visitors are split evenly between the two.

Worked example

Say the current conversion rate is 2% and the smallest relative lift you want to detect is 20%: the target rate is 2.4%. The formula gives 21,109 visitors per variant; 42,218 for the two together.

If 20,000 visitors a week enter the test, 42,218 ÷ 20,000 = 2.11; the test runs for 3 full weeks.

How to read the result

  • The number is set before the test starts, and the result is not read until you reach it. Looking at interim results and stopping early makes it more likely that you declare a winner where there is no real difference.
  • Detecting a smaller difference takes far more visitors: halving the lift you want to detect roughly quadruples the sample you need.
  • Days of the week behave differently, so a test runs in full weeks. If the result says months, testing this effect on this page is not realistic; try a bigger change or a page with more visits.
  • 80% power means that a real difference of this size can still be missed in one test out of five.

Questions

The questions that come up most about this calculation. If yours is not here, write to us and we will answer within one working day.

How many visitors do you need for an A/B test?

It depends on your current conversion rate and the difference you want to detect. The lower the rate and the smaller the difference, the faster the number grows. For example, detecting a 20% relative lift on a 2% rate at 95% confidence and 80% power takes about 21 thousand visitors per variant.

How long should an A/B test run?

Until you reach the total number of visitors you need, and always in full weeks. Weekday and weekend visitors behave differently, so a test that ends mid-week can mislead. Even with plenty of traffic, it is sound to run for at least one full week and preferably two.

What do 95% confidence and 80% power mean?

95% confidence limits to 5% the chance that the test reports a difference when the two versions do not really differ. 80% power means that if a difference of the size you set really exists, the test will detect it with an 80% probability. Both are values in common use for A/B tests.

Related guides

Tell us about
your project.

Fill in the form or write to us directly, and we will agree on a day to talk.

Your first meeting is with Deniz Sezer, founder of Detmore.

First meeting free · Reply within one working day · Written proposal in 2 to 3 days · NDA ready

What do you need? pick as many as you like

Name, email and phone are required.

Contact

We reply within one working day.