Short answer
An email A/B test sends two versions to part of your list and compares one element only: the subject line, sender name, content or send time. Do not let open rate pick the winner: Apple’s Mail Privacy Protection can inflate opens. Judge by clicks and orders instead.
Picture a Monday at Örnek Çay Evi, a fictional online tea shop with a few thousand people on its list; every example in this article is made up. The marketer writes two subject lines, switches on the tool’s A/B test and, at the same time, changes the sender name too. An hour later B’s open rate looks higher, so the tool sends B to the remaining recipients. By the weekend the number of orders has not moved. How to A/B test emails is the mirror image of the three mistakes in that scene: change one element, wait long enough and pick the winner with the right metric.
We covered the general logic of testing (hypothesis, sample size, not peeking) in the A/B testing guide, and how to write the subject line itself in the subject line guide. This article covers only the email-specific part: how tools split a list, how they pick the winner, which metric to trust and what to do when your list is small. The tool details come from the Mailchimp, Klaviyo and HubSpot help pages. If you are still choosing a tool, see Klaviyo or HubSpot.
1. Decide what to test
Mailchimp’s A/B test page lists four variables for an email test: subject line, From name, content and send time. The table below shows what each variable answers and the examples the tool pages give.
| Variable | Question it answers | Example on the tool pages |
|---|---|---|
| Subject line | Which wording stands out in the inbox? | Klaviyo: length, personalisation, emoji, statement or question |
| Sender name | Is a person’s name or the brand name better recognised? | Klaviyo: company name versus a more personal name; HubSpot: an employee’s address versus a generic department address |
| Content | Which layout, image or call to action gets clicks? | Klaviyo: button size and colour, call-to-action text, template layout |
| Send time | Which day and hour get a response? | Mailchimp and Klaviyo: this test goes to 100% of recipients |
We suggest starting with the subject line (this is our advice): it changes a single field, and it is easy to apply the winner to the next send.
2. Change only one variable at a time
Klaviyo’s campaign A/B test page says to test only one variable at a time. Changing several together makes it difficult, or impossible, to tell which one made the difference. The same page says that if you are testing the subject line you should not change anything else, and that in a send-time test the content and subject lines of both versions must stay identical.
In the Örnek Çay Evi scene the marketer changed the subject line and the sender name together. Even if B wins, you cannot know whether the subject or the name did it. Before the test starts, write the hypothesis in one sentence: “If the subject line names the product, clicks go up.” If the sentence holds two elements, split it into two tests.
Keep the number of versions low as well. Mailchimp allows up to three versions of one variable, while Klaviyo recommends only two. Every extra version cuts the test group into smaller pieces. Our advice is two versions.
3. Know how the tool splits the list and picks the winner
The three tools follow the same pattern. A percentage of recipients forms the test group, and the group is divided between the versions. When the waiting time ends, the winning version goes to everyone else. The figure below shows the flow.
The details differ by tool:
- Mailchimp: you must send the test to at least 10% of recipients. The winner can be chosen by open rate, click rate, total revenue or by hand. A send-time test always goes to 100% of recipients.
- Klaviyo: you set the test size and the duration. In the page’s example, a 40% test group gives 20% of recipients to A and 20% to B, a winner is chosen after 6 hours and the rest receive it. The winning metric can be open rate, click rate or placed order rate. A send-time test goes to 100%.
- HubSpot: you choose the percentage of recipients in the test, the winning metric (open rate, click rate or click-through rate) and the test length in hours. You also pick a fallback version that is sent if the result is inconclusive.
On timing, Mailchimp recommends waiting at least 4 hours after sending before the winner goes out. It also says that if none of the recipients open or click any version, the final set of recipients gets the last version by default. So do not trust the automatic winner on a very small test group. Choosing the wait according to when your readers usually read email is our advice: a few hours may miss people who read later in the day.
You can also run the test on a single segment instead of the whole list; Mailchimp asks you to choose an audience or segment at the recipients step.
4. Do not let open rate pick the winner
Apple’s Mail User Guide says remote content in incoming messages can let a sender collect information such as when and how many times you view a message, and that Mail Privacy Protection stops this. With “Protect Mail Activity” selected, remote content is downloaded in the background when the message is received, not when it is viewed. So an open counter can go up even if the reader never looks.
Klaviyo ties this directly to A/B tests. Its statistical significance page says that Apple’s automatic opens are counted when open rate is the winning metric, and that statistical significance does not show that recipients opened the email on purpose. It also suggests considering click rate if open-rate results are hard to interpret. Its campaign page suggests a custom report that includes an MPP property if you test regularly and more than about 45% of your opens come from Apple Mail. That page also recommends open rate for subject line, preview text and sender name tests; read that recommendation together with the Apple warning on the same page. Mailchimp tells you to pick the winner by click rate, not open rate, when you test content, because opens happen before the reader sees the content.
Our advice is in the table below. It is not the tools’ official recommendation; it is our reading of the pages above.
| What you test | Winning metric we suggest | Note |
|---|---|---|
| Subject line | Clicks; orders if possible | Record opens too, but do not decide on them |
| Sender name | Clicks; orders if possible | Same reasoning |
| Content | Clicks; orders if a store is connected | Mailchimp also recommends clicks for this test |
| Send time | Clicks and orders | It goes to everyone, so the result guides the next send |
On the order side, Mailchimp’s option is total revenue; in Klaviyo, placed order rate can be chosen only if the account has an order metric. Without these, add UTM tags to your links and watch what visitors do in your analytics tool. Clicks are rarer than opens, so differences are harder to see on a small list; the next step deals with that. The open rate term is explained in the glossary.
5. Read the result carefully if your list is small
The numbers on the help pages differ a lot, because each tool uses its own criteria:
| Tool | What the page says |
|---|---|
| Mailchimp | The test group must be at least 10% of recipients. It recommends at least 5,000 subscribed contacts per combination for the most useful data, and says smaller tests can also teach you about your audience’s preferences. |
| Klaviyo | For a campaign, a result counts as “statistically significant” when each variation has more than 50 successful deliveries and the win probability is above 90%. For a “promising” result it advises repeating a similar test. |
| HubSpot | For best results it recommends sending an A/B tested email to at least 1,000 contacts. |
Klaviyo’s page also says that a significant result does not guarantee the same outcome with a different audience, message or send time. Everything beyond this is our own view. If your list has a few hundred people, a difference in a single send is very likely chance. In that case:
- Raise the test to 100% (Mailchimp and Klaviyo allow it). Half the recipients get A and half get B; with no recipients left, no winner is sent and you carry what you learned into the next send.
- Repeat the same idea over several sends. Turn a difference into a rule only if it repeats in the same direction.
- Use the A/B test sample size calculator to see the number you need for your own rates.
6. Record the result
When the test is over, nobody will remember which version was which. Keep these in one place for every test:
- Date, list or segment, and the number sent.
- The one-sentence hypothesis.
- The single element that changed and the text of both versions.
- Test group share, waiting time and winning metric.
- For each version: delivered, clicked and ordered; note opens separately.
- The label the tool gave (for example “promising” in Klaviyo) and your decision.
- The next test.
A made-up example: “Hypothesis: if the subject line names the product, clicks go up. 900 delivered each for A and B; A had 27 clicks, B had 36. Tool: promising. Decision: do not make it a rule, repeat in the next send.” Log flat and losing results too. They stop you testing the same idea again two years later.
In Detmore’s email and CRM service, subject, timing and content tests run every month, and open rate alone is never the measure.
Common mistakes
- Changing the sender name or preview text together with the subject line.
- Picking the winner after one hour on open rate alone.
- Trusting the automatic winner on a very small test group.
- Turning one send’s difference into a general rule such as “always use emoji for this list”.
- Not recording the result and running the same test again.
Common questions
How big does a list need to be for an email A/B test?
The tools give different numbers. Mailchimp recommends at least 5,000 subscribed contacts per version, HubSpot at least 1,000 contacts and Klaviyo looks for more than 50 deliveries per variation and a win probability above 90%. If your list is smaller than that, do not treat the result as settled, and repeat the test.
Which metric should pick the winner in an email A/B test?
We suggest clicks and, if possible, orders. With Apple’s Mail Privacy Protection on, remote content is downloaded when the message is received, so the open count may not reflect real reading. Klaviyo also says to consider click rate when open-rate results are hard to interpret.
How long should an email A/B test run?
Mailchimp recommends waiting at least 4 hours before the winner is sent. For anything longer, we suggest basing it on when your readers usually read email. A send-time test goes to all recipients, so you use its result in the next send.
One thing you can do alone today: open the latest campaign report in your tool and write the open, click and order numbers side by side, then pick one variable and one winning metric for the next send. If your tool is set to pick the winner by open rate, change it to clicks.
Sources
This article is general information, not legal, financial or technical advice. Platforms, their rules and the law change over time, so some details may have changed since 15 November 2026 or contain small errors. Before you make a decision for your business, check the current official sources or write to us.