fbpx What is A/B testing? - Bang Consulting Bang Consulting

Check out our blog for hints, tips and news about all things marketing…

What is A/B testing?

A B testing

We have said before marketing is not an exact science, and neither are there any guarantees. Every business is different, and this is where the beauty of A/B testing comes in.

What is A/B testing?

In a nutshell, it is a method of testing where two versions of something are compared and assessed to see which is the most popular/successful. This is very common in website design/development and digital marketing.

The key is to only test one thing at a time. So, if we take email marketing, you might test two different subject lines, two different times of day, or even two different days in the week.

If you use a system to manage the test, such as MailChimp, in the instance of email marketing, it will only test on a percentage of your database, and then send the remainder of the database the most successful result.

Having recently read an article on the MailChimp database about the success of the rather bland sounding Newsletter August 20xx, as opposed to a more exciting (supposedly!) subject line, we decided to test this for ourselves.

So, we ran an A/B split test for a client

The client had 1120 people on their database, so MailChimp selected 226 emails to send the test. Half would have the subject line: [Company Name] Newsletter Xxxxxx 20xx, and the other half would have a subject line playing on the name of the product being promoted.

The newsletter subject headline won – in terms of open rate, number of times opened, average number of times opened and click through rate. The system then sent the “winning” subject headline with the email to the remainder of the database.

Now, before you leap in, there are some words of caution here:

  • The difference between the two open rates was marginal – 29 recipients for one, 26 for the other. We also believe the client was one of the recipients, which may have further skewed it, as he is likely to have opened the mail several times, and possibly forwarded it.
  • 226 is not a particularly big number for testing, and clearly the bigger the number the more information you are getting. This may sound confusing, but what we mean is, with a low number 1% may represent one or two people, whereas with a bigger number 1% may represent five or ten people, or more, so is much more meaningful.
  • One test is not enough – you need to test the same thing again, to make sure it was not a one off.
  • You should also test different aspects – such as the colour of your call to action text, whether asking people to ring or email changes the response etc
  • Running a test using a system is a lot easier than trying to do it manually – and there’s a lot less brain ache required!

We realise this is a little more in depth than some of our Marketing Phrases, but thought you would be interested to read a real life example, rather than just a theory.

We definitely advocate testing your own marketing – and not just online. This works just as well with offline marketing too – does a red flyer get a better response than a blue flyer etc…

Don’t get so caught up in this that you distract yourself from what you are trying to achieve, but equally learn what works, hone it, refine it, and then do more of it!

If you would like to get your marketing sorted, then give us a call on 01256 83 11 10.