Why you need to be A/B testing your email marketing and how to get started

Why you need to be A/B testing your email marketing and how to get started

Charlotte Jenkins

Charlotte Jenkins
14th August 2019

Email marketing is a proven way of generating leads and encouraging repeat business. Your audience are already invested because they’ve left you their email address, and some may have even purchased from you already.

And with the availability of time-saving, free and intuitive email marketing software, you get a better ROI than any other marketing tactic.

But businesses who are using email marketing but aren’t A/B testing their campaigns are missing out on an invaluable opportunity to make more of their campaigns. In this post, I’ll explore A/B testing and how you can get started straight away.

What is A/B testing?

A/B testing is another way of saying “split testing”. Split testing is when you run two campaigns concurrently, with a variable changed on the second campaign, so that you can test the effect that variable has had.

When we talk about A/B testing in email marketing, we mean we’re testing the original version of an email against another version with one single element changed. This element might be a subject line, call-to-action, image, from field, or content.

It’s important that only one element is changed otherwise you won’t be able to identify which variable is having the impact on the campaign. Everything else, including send time, needs to be exactly the same, or the test is void.

If you do want to test more than one element, this is called multivariate testing, and is something slightly different.  

How A/B testing works

Your email marketing tool will, more than likely, have in-built A/B testing features available to you.

It’ll help you set up your campaign from start to finish, creating two identical emails, apart from one variable you’d like to test.

Then, a little while after you’ve sent your campaign, you’ll receive a report to show you which email was most successful (the report will look at open rate, click-through rate and conversion rates).

You are then able to make a decision as to whether your changed variable has been a success, and whether or not you think it would be worth adopting in future campaigns.

The benefits of A/B testing

Businesses do A/B testing simply because they want to improve the effectiveness of their email marketing campaigns. Essentially, it’s a conversion optimisation tool, making sure emails are honed as best they can for achieving whatever it is they were originally designed to do.

For example, your email campaign may be focused on getting people to click through to your website. As such, your A/B test might want to try out a new (bigger, brighter, more obvious) call-to-action to see if more people will click on it through to your website.

Email marketing, when used well, is the most cost-effective sales and marketing channel at your disposal. But it’s more effective if you can gather data from testing to reach the right audience, at the right time, with the right message. Split testing can help you do just this, helping to understand how your audience best engages with your emails.

A/B split testing, in short, will help you to:

  • Know which campaigns work and which don’t, based on your customers’ real-life behaviour
  • Backup your email marketing “hunches” with real data
  • Design and create effective and targeted email marketing campaigns so they reach the right people, at the right time, with the right message
  • Ultimately increase your email open, click-through and conversion rates

How to A/B test

Before you get started, you first need ask yourself a few questions:

What one thing am I testing?

Here you need to identify which variable you want to test. It might be any of the following:

  • Subject line
  • From name
  • Salutations (e.g. using first name, Mr/Mrs/Ms + surname, or no salutation)
  • Email copy
  • Email design
  • CTA design/colour/positioning
  • CTA wording

How will I measure success?

You’ll be limited to what your email marketing tool will allow you to measure, but as a rule (and based on Mailchimp as a guide), you’re usually able to measure:

Open Rate

This is the number of recipients who open your email.

Your open rate is usually a good test of both subject lines and from fields. It’s these elements your recipients will be able to see in their email preview from which to decide whether or not to open the email.

Click-through rate

Your CTR determines how many people have opened your email and go on to click on your call-to-action.

This tests how strong your CTA is, whether that be its positioning, colour, font, wording or size.

It’s worth noting that if you have more than one CTA in your email, make sure you only test one at a time.

But it’s also a good gauge of your email’s wording. There’s little point including a CTA in isolation without some text explaining why it should be clicked on.

Total revenue (if you’re connected to your online store)

This determines how many people make a purchase as a result of your email. They will have to have opened your email, clicked through to your website and then gone through to your checkout.

As such, this is a good test of your email copy – you might be drawing attention to a new product or offering a discount or special offer – as well as your CTA.

But it also tests what’s going on beyond your email. Once people have clicked through from your email, they need to arrive at an optimised landing page which encourages them to make a purchase.

Is my test statistically significant?

To ensure your A/B test has meaning, it needs to be sent to a big enough pool of recipients to demonstrate any significant result. Mailchimp, for example, advises that A/B tests are sent to a mailing list of at least 5,000 in each combination.

Setting up A/B testing in Mailchimp

Building your A/B test in Mailchimp is very straightforward, because it follows a series of simple steps.

Create your campaign

1. First go to the Campaigns page and select Create campaign.

Create campaign screenshot

2. Within the popup, click Email.

Email screenshot

3. Click A/B Test.

A/B Test screenshot

4. Now enter your campaign name and click Begin.

Choose your recipient list

You can either build a new segment or choose an already saved segment.

Selecting your variables

Mailchimp gives you a choice of four variables: Subject Line, From Name, Content and Send Time.

1. Choose your variable by clicking the plus sign on the relevant box.

2. Now click the plus or minus sign to choose how many variations of the variable you want to test. You can only select one variable but can choose up to three variations of that variable. So, for example, if you want to test your send time, then you can choose up to three different send times to test.

What would you like to test screenshot

3. Now you need to choose the percentage of your recipients you want to send your test combinations to.

Test combinations screenshot

4. Now you need to choose how to measure the winning combination, either by click rate, open rate, or revenue. Alternatively, you can choose to manually choose your winner, which will mean you’ll need to go to the Reports page at the end of your test duration and pick your own winner. You also need to input the duration of your test.

Setting up your emails

The setup process will, of course, depend on which variable you’ve chosen. I’ll take you through the process from the perspective of an email subject test as an example.

1. Type in your subject lines in the fields provided.

Email subject screenshot 2. Check the boxes next to your preferred settings or tracking options to include. Track opens and Track clicks can’t be disabled, since they’re used to determine the winner of the test.

Tracking screenshot

Creating your email content

If you’re testing a subject line, from field or send time, then your content step will be straightforward. You simply need to input your content and then skip through to the Confirm step.

If you’re testing content:

1. Click Add content

Content setup screenshot

2. Choose your layout from the Templates available.

3. When you get to the Design step, design and test your campaign. When you’re satisfied, click Next.

4. On the Description step, type in your description to describe the variation of your content (this will be seen only by you so you can differentiate your two variations).

5. Click Continue to content to get back to content variations.

6. Repeat steps one to five until you’ve completed each of your content variations.

7. When you’re happy with everything, click Next.

Confirm your campaign

Once you’ve previewed your emails and gone through the pre-delivery checklist, you’ll be ready to schedule your campaign.

When you’re ready to send, click Send or Schedule, depending on whether you want it to send immediately or at a predefined time.

Schedule screenshot

Best practices and top tips

Here are some top tips for getting your A/B testing right:

1. Create a hypothesis

Start with a premise that you want to test e.g. “if I include a first name in the subject line, I’ll increase the open rate”.

2. Know what to measure

If you know what you want to improve, open rate, click-through rate, or revenue, then you’ll need to work backwards to determine which variable you need to test.

As mentioned above, open rate is affected by subject line and from field, click through by CTA and body copy, and revenue by body copy, CTA and landing page optimisation.

3. Choose just one element

Choose more than one element and you need to create a multivariate campaign.

4. Make your test statistically significant

To give your test a decent chance, you need to include at least 5,000 recipients if you can. If you don’t have that many contacts then include as many as you can.

5. Keep every other variable exactly the same, including send times

Your test results won’t give you an accurate result if you have changed more than one element. This means that everything else – including the send time – must be exactly the same.

In summary

The power of email marketing is undeniable. And if you’re already using it to grow your business, then you’re onto a good thing.

But you can go one step better using A/B testing. Most email marketing tools make it easy for you to set up, and the results can really help you to optimise your campaigns to improve your reach, website traffic and, ultimately, revenue.