Email A/B Testing for Ecommerce: What to Test and How to Learn Faster

A/B testing is easy to make look sophisticated.

Change a button. Change an emoji. Change the subject line. Run a test. Declare a winner. Move on.

The problem is that most ecommerce email tests never change the strategy. That makes them activity, not learning.

A useful testing program should answer questions that matter. For example:

• Should we send this flow sooner?
• Does this audience need a discount?
• Does plain text outperform designed creative here?
• Should product recommendations be automated or curated?
• Does this popup offer create more customers?
• Does a shorter welcome series convert better?

The best test is not the cleverest test. It is the one where the answer changes what you do next.

By Emiel Dingemans, Founder of Magicianly

What Is Email A/B Testing?

Email A/B testing is the process of showing two controlled variations of an email, flow, signup form or strategy to comparable audience groups to determine which version produces the stronger outcome.

The important words are controlled and outcome.

If multiple major things change, you do not know what caused the result. If you measure the wrong outcome, you may optimize the wrong behavior.

Start With a Hypothesis

Do not begin with “what can we test?” Begin with “what do we believe is currently limiting performance?”

Then write it down: if we change X, then Y should improve because Z.

For example: if we send the first abandoned cart email after 30 minutes instead of four hours, we expect more recovered carts because purchase intent is still fresh. Our guide to abandoned cart timing covers that specific question in more depth.

Now the test has a reason.

The Magicianly Testing Loop

Use this framework:

1. 1hypothesis;
2. 2test;
3. 3result;
4. 4learning;
5. 5next decision.

The final step is the most important. A result should influence future copy, timing, creative, segmentation, offer or flow structure.

If nothing changes, ask whether the test mattered.

Prioritize High-Volume Areas

One of the biggest mistakes is spending weeks testing a flow that triggers 40 times per month, while the welcome series, cart, checkout, post-purchase and signup form are processing thousands of customer actions.

A small improvement in a high-volume lifecycle moment can matter much more than a large percentage lift in a niche flow.

Prioritize tests using potential impact, multiplied by traffic, multiplied by confidence, weighed against effort.

What Should You A/B Test in Email Campaigns?

Subject Line

Useful when enough recipients exist. Measure more than open rate. A subject line that gets more opens but fewer orders is not necessarily the winner.

Plain Text vs Designed

Especially useful for founder messages, deadlines, product announcements and educational campaigns. Our comparison of plain text and designed emails explains where each format tends to earn its place.

Offer Framing

Percentage off, dollar amount, bundle, free gift or mystery offer.

Campaign Angle

For example, product features against a customer story.

CTA

Use when CTA clarity is a plausible constraint.

Audience

For example, a 90-day engaged audience against a broader engaged audience. Be careful here: audience tests can affect deliverability and total revenue, so interpret the result in the context of your wider campaign strategy.

What Should You Test in Flows?

Flows offer strong testing opportunities because they operate continuously. Useful variables include:

• timing;
• email count;
• message angle;
• offer;
• first-time versus returning split;
• social proof;
• plain text versus designed;
• risk reversal;
• cross-sell;
• product recommendation method.

Test Timing

Timing should reflect intent. For example:

Cart abandonment

30 minutes against 60 minutes.

Checkout abandonment

15 minutes against 45 minutes.

Welcome series

Same-day second email against next-day, which is one of the decisions we cover in the welcome email series guide.

Post-purchase

Education before delivery against after delivery.

Do not change timing and content simultaneously.

Test Offers Without Destroying Margin

A higher discount can create more orders. That does not automatically make it the winning test.

Compare conversion, revenue, margin, average order value and downstream customer behavior.

If 20% off generates more attributed revenue than 10% off but destroys incremental profit, the result is far less exciting than the dashboard suggests.

Test When Discounts Are Introduced

Sometimes the better test is not 10% against 15%. It is discount now against persuasion first.

In an abandonment flow, variant A offers an immediate discount while variant B runs reminder, trust, objection handling and only then an incentive. Our abandoned cart email guide uses that structure as the default.

This can tell you whether price is actually the constraint.

Test Product Recommendations

Useful comparisons include automated recommendations against manual selection, best sellers against complementary products, and category-based against purchase-based logic.

Do not assume machine-generated recommendations are always better. Poor product taxonomy can produce weak output. Let the account data decide.

Test Signup Forms

Signup forms often have enough traffic to create meaningful learnings quickly. Useful tests include:

• offer;
• mystery versus explicit offer;
• popup timing;
• scroll threshold;
• question or micro-commitment;
• CTA;
• full-screen versus alternative layout;
• SMS step;
• teaser behavior.

Measure beyond signup rate. Also evaluate welcome series conversion, first purchase, unsubscribe behavior and revenue per visitor, as covered in our Shopify email popup guide.

Do Not Test Five Things at Once

Imagine variant A with a 10% discount, a blue button, plain text and a 30-minute delay, against variant B with a free gift, a red button, a designed email and a two-hour delay.

Variant B wins. What did you learn? Almost nothing. The winner may have come from the offer, the creative, the timing, the CTA or the interaction between them.

Keep tests interpretable.

What Metric Should Decide the Winner?

That depends on the test. Subject line tests can be diagnosed with open rate. CTA tests can use click rate. Offer tests need purchase and margin. Popup tests need more than signup rate. Flow timing tests need order conversion.

Do not choose the metric because it moves fastest. Choose it because it matches the hypothesis, and keep the retention metrics that matter as the ultimate reference point.

Be Careful With Small Samples

Small audiences create noisy results. That does not mean every ecommerce brand needs a statistics department.

It means you should not make major strategy changes because variant A got four orders and variant B got six. Treat low-volume results as directional. High-volume tests deserve more confidence.

Do Not Let the Platform Pick the Winner Too Early

Some platforms offer automatic winner selection. That can be useful, but be cautious when volume is low, test duration is short, the commercial metric takes time to appear, or the platform optimizes for the wrong metric.

If necessary, run controlled splits manually and review the result yourself.

Document Every Test

Use a simple structure: brand or area, test, hypothesis, result, learning, next step, plus an optional screenshot or data export.

The goal is organizational memory. Six months from now, the team should still know what was tested, why, what happened and what changed.

Losses Are Useful

A failed test can be extremely valuable. If you expected a discount to increase conversion and it did not, you learned that price may not be the main objection. That can change future strategy.

Do not only document winners. Document wins, losses and inconclusive results.

Do Not Test for the Sake of Testing

A testing quota creates pointless experiments. The objective is not running 20 tests this month. It is learning something that changed how the account operates.

Testing should compound knowledge.

What Should You Test First?

Start with:

1. 1high-volume flows;
2. 2signup forms;
3. 3high-frequency campaigns;
4. 4offers;
5. 5timing;
6. 6major message angles;
7. 7segmentation;
8. 8creative format.

Leave minor cosmetic tests for later unless there is a clear reason to believe they matter. If you are not sure where the constraint sits, start with an email marketing audit.

The Best Testing Program Gets Smarter Over Time

Every useful test should make the next test better. Over time you discover which offers work, which objections matter, which products resonate, which audiences are valuable, which timing works and which format fits the message.

The account becomes less dependent on generic email advice and more grounded in what your customers actually do. That is the point of A/B testing.