What is an A/B test, and how to run one

From deciding whether a test is the right tool to the sample size, from not peeking to reading a flat result: how to run a single test from start to finish.

· 11 min readData

Short answer

An A/B test shows two versions of a page to random visitors in the same period and checks which one does better on a single primary measure. To run it properly, fix the hypothesis and sample size before starting, do not judge before the number is reached, and write the result down even when it is flat.

A new version of a form page goes live, and on day four the dashboard shows version B thirty per cent ahead. Someone says “it won”, the test is closed and the new version goes to everyone. Two weeks later the overall conversion rate is back where it was, because the gap in those first days was luck. Half of the answer to “what is an A/B test and how do you run one” sits in this scene. Setting up a test is easy; the hard part is running it properly to the end, against the urge to look at the result.

An A/B test shows two versions of a page or element to random visitors in the same period and checks which one does better on a single measure chosen in advance. For a short definition see the A/B test entry in the glossary. How to find what to test, and the rest of the conversion work, is covered in how to increase your conversion rate. Here we stay with a single test: setting it up, running it and reading it. If you want to split traffic from your ads between two versions, see how to set up Google Ads experiments.

A test has three stages. Most mistakes happen when the work of one stage is left for the next.

The three stages of an A/B test: what to do before starting, while it runs and when it ends 1 Before One hypothesis One primary measure Guardrail measure Sample size 2 During Random split No other changes No peeking Full weeks 3 After Read at the number Log a flat result Record the decision Remove test items Test log: hypothesis, duration, counts, decision The three stages of an A/B test: what to do before starting, while it runs and when it ends 1 Before One hypothesis One primary measure Guardrail measure Sample size 2 During Random split No other changes No peeking Full weeks 3 After Read at the number Log a flat result Record the decision Remove test items Test log: hypothesis, duration, counts, decision
Figure: The work of a test falls into three stages; hypothesis and sample size come first, not peeking applies while it runs, and reading and logging come last.

1. First decide whether a test is the right tool

A test suits a page with traffic and a single change whose effect you want to measure. If the page gets few visits, catching a small difference takes months; heatmaps, session recordings and a few conversations with users will point the way faster than a test. If the problem is plainly a bug (a button that does not work, a broken form), you do not test it, you fix it. Do not guess the threshold: do the calculation in step 3 below, and if the duration comes out in months, the tool is wrong. Also make sure that what you want to measure can come down to one number; a question like “does the brand look more trustworthy” does not fit an A/B test, and a user interview answers it better.

2. Choose one hypothesis and one primary measure

A hypothesis is one sentence: “We saw this; if we change that, this behaviour changes and this number goes up.” The primary measure is the number in that sentence, and you choose it before the test: form submissions, orders, appointment requests. When the test ends you will have several numbers, and it is quite likely that one of them looks “good”; if you did not write the primary measure down first, you pick that number and build the story afterwards.

Say an appointment form has seven fields and most people who abandon it leave at the third. The hypothesis reads: “If we cut the fields to four, form submissions go up.” The primary measure is form submissions; the guardrail measure is the share of incoming requests the team considers useful, because a shorter form can bring more requests that are less qualified.

A guardrail measure is a number that must not get worse while the primary measure improves: returns, cancellations or page load time. And one more rule, one change only: if you change the headline, the image and the button together, you will not know why the winner won, even if there is one.

3. Work out the sample size before you start

You need three things here: the current conversion rate, the smallest difference you want to catch, and how reliable you want the test to be. In plain words, 95% confidence limits to five per cent the risk that the test says “there is a difference” when there is none. 80% power is the chance that the test catches a difference of that size when it really exists. Both are commonly used values, and the A/B test sample size calculator takes them as fixed.

A round example: your page converts 2% of visitors and you want to catch at least a 20% relative increase, that is 2.4%. The calculation gives about 21,000 visitors per variant, about 42,000 in total. If 20,000 visitors a week enter the test, that is three full weeks. If you halve the difference you want to catch, to 10%, the number needed rises roughly fourfold. A small difference needs a large sample; that is why testing a small cosmetic change on a low-traffic page is a hopeless task.

4. Split the traffic and do not peek

Visitors are assigned to the two versions at random, usually in equal shares, in the same period. That spreads outside effects such as season, campaigns or news evenly across both sides; so comparing last month’s page with this month’s page is not an A/B test. A returning visitor should see the same version again. After the start, check that the numbers landing in the two arms are in the ratio you expected; if the split is far from even, something is wrong with the setup and the result cannot be trusted.

Then keep your hands off. If you look every day, you raise the chance of catching a day when luck has briefly put one arm ahead; that is the mistake in the opening scene. The reason is simple: early on, with a small sample, the gap between the arms swings widely, and every look gives that swing one more chance to mislead you. The rule is simple too: do not decide before you reach the number you calculated. Do not run a campaign or a design update on the same page during the test either; it makes the result unreadable.

5. Measure the duration in full weeks

Weekday and weekend visitors behave differently, so a test runs in full weeks: if the calculation gives 2.1 weeks, it is rounded up to three. The duration ends when the visitor number is reached, not by a fixed rule such as “two weeks is enough”. On the other hand, do not keep it longer than needed: Google wants a test to run only as long as necessary (see the search section below).

6. Read the result, a flat one too

Back to the earlier example: 21,000 visitors in each arm and 420 conversions in arm A (2.0%). The result can come out in three ways (the conversion counts are round, illustrative figures):

Conversions in arm B What it says What you do
504 (2.4%) The gap is 84 conversions and is very hard to explain by chance Ship B, check the guardrail measure, log it
441 (2.1%) The gap is tiny; the true difference could be anywhere from about minus 0.2 to plus 0.4 points Do not declare a winner; log it and design a bolder change
336 (1.6%) B is clearly lower; a gap of 84 conversions is very hard to explain by chance Do not ship B; compare the cause with the hypothesis and write it down

A flat result is not a failure. It means “this test could not catch a difference of this size”: either the change does not work or its effect is smaller than you planned for. In the second case you may still keep the new version if it is simpler; but writing “B does not work” would claim more than the evidence says. If the primary measure is up but the guardrail measure got worse, for example orders rise and so do returns, reassess the gain. And once the test is over, do not trawl the result through a thousand breakdowns (device, source, country): every slice you look at raises the odds of “finding” a difference by chance. Use a breakdown if the hypothesis named it in advance; otherwise make it the subject of a new test.

7. The search side: what Google says

Google Search Central’s page on A/B testing sets a few rules so that tests do not harm search results:

  • Do not cloak. Showing Googlebot one version and people another goes against the spam policies, even for a test. If you control the test with cookies, Googlebot generally does not support cookies, so it sees only the version that visitors without cookies see.
  • If you use separate URLs, put rel=canonical on the alternate URLs pointing to the original URL.
  • If you use redirects, choose a temporary (302) rather than a permanent (301) redirect; JavaScript-based redirects are also fine.
  • Run the test only as long as necessary. When it ends, make the winning version permanent and remove alternate URLs, test scripts and markup as soon as possible. Google says it may treat an unnecessarily long test as an attempt to deceive search engines.

The same page adds that small changes such as the size, colour or placement of a button, or the call-to-action text, can have a surprising effect on how users interact with a page. It says such changes often have little or no impact on the page’s search snippet or ranking.

8. Write the test down

When the test ends, nobody will remember that page. For each test, one place should hold:

  1. The hypothesis, the primary measure and the guardrail measure.
  2. The calculated sample size, and the start and end dates.
  3. Visitors and conversions in each arm.
  4. The decision and its reasoning, including for flat and losing tests.
  5. The next step.

Losing and flat tests are the most valuable records: they stop you testing the same idea again two years from now. The Monday report is a good place to read a few chosen numbers each week; and to be sure the measurement itself is set up correctly, see the GA4 setup checklist.

Common mistakes

  • Declaring the leading arm the winner on day four.
  • Choosing the primary measure after the test is over.
  • Testing a subtle change on a page without the traffic for it.
  • Running a campaign or a design change on the same page during the test.
  • Moving on from a flat result with “it didn’t work” and no entry in the log.
  • Leaving alternate URLs and scripts on the site once the test is over.

Common questions

What is an A/B test?

An experiment in which two versions are shown to random visitors in the same period and compared on one primary measure. Because random assignment spreads outside effects evenly across both sides, it lets you say the difference comes from the change.

How long should an A/B test run?

Until it reaches the number of visitors you calculated in advance, and always in full weeks. If that number comes out in months, move ahead with behaviour data on that page instead of a test. Google also wants a test to run only as long as necessary.

Can an A/B test hurt SEO?

Generally not, if you follow the rules: do not cloak, use rel=canonical on separate URLs, choose 302 for redirects and remove the test elements when it ends.

The one thing you can do today after reading this: turn your next test idea into a one-sentence hypothesis, write down its primary measure and enter your current rate into the sample size calculator. If the duration it gives is not reasonable, that idea is not a test yet, it is an item for the to-do list. If you would like us to set up the measurement and the test routine from the outside, that is part of our data and CRO service.

Sources

This article is general information, not legal, financial or technical advice. Platforms, their rules and the law change over time, so some details may have changed since 11 November 2026 or contain small errors. Before you make a decision for your business, check the current official sources or write to us.

Tell us about
your project.

Fill in the form or write to us directly, and we will agree on a day to talk.

Your first meeting is with Deniz Sezer, founder of Detmore.

First meeting free · Reply within one working day · Written proposal in 2 to 3 days · NDA ready

What do you need? pick as many as you like

Name, email and phone are required.

Contact

We reply within one working day.