How to do an A/B test

If you’ve ever needed to test the performance of a new piece of content or design on a page of your site, chances are you’ve come across the term A/B testing.

For anyone working in marketing, A/B testing is an extremely valuable method to try out changes to areas of your site and gauge which version is the most effective for your users. It’s also the perfect tool to use if your conversion rates are a little lacklustre and you want to find a way to drive them up. There are many online tools to help you conduct an A/B test. In particular, we’ll take a look at how you can use Treejack for A/B testing.

What is A/B testing?

A/B testing, also known as split testing or a split run, is a method used to compare two versions of the same thing in order to test the effectiveness of each variation. Traffic is also split 50/50 between each version in order to gather data. A/B testing is usually conducted on a page of a site, but also extends to emails and apps or anything really.

A/B testing isn’t anything new, and the things you test don’t even have to be online. In fact, even medical testing for drugs is a form of A/B testing. One group would be given one type of drug to trial, while the other group would be given a placebo. You could even test two types of packaging for a particular product and see which performs best.

In an online setting, the types of content people test is extremely varied. Some of the many testing examples include:

• Call to actions (CTAs) e.g., buttons or banners that invite people to click them or take action
• Headlines
• Images
• Display ads
• Layout
• Information architecture
• Forms
• Email copy
• Landing pages (pages on your site that people “land” on, like an entry point)

A/B testing is a great way to systematically improve conversion rates. In fact, the Conversion Rate Optimization Report 2015 from Econsultancy and Redeye shows that 58% of businesses surveyed use A/B testing as a way to improve conversion rates. Additionally, 72% of businesses focus their testing to gauge the effectiveness of their landing pages.

Although it sounds like something that should only be carried out when you want to make changes, A/B testing can actually benefit your business if it’s done on a regular basis. You might be familiar with the old adage “test early, test often” — the same saying rings true for A/B testing.

When performed consistently, A/B testing can actually show you which marketing strategies are working for your business and which aren’t. This gives you hard and fast data to help inform any decisions you make regarding strategy or even the structure of your site.

Optimizing a multi-step process even by just a little bit can lead to big changes down the line. For example, let’s say you had a four step process and 1,000 leads in the first step. Each step after that was a 50% take (e.g., 1,000 > 500 > 250 > 125). If you lifted each step by 5%, you could end up with an overall 25% improvement. Any increase, no matter the size, is definitely helpful.

The test process

A/B testing is a simple concept that’s easy to understand and carry out. However, it can be hard to do it well. Think of it like an iceberg: The easier tasks, like changing the colors or headings on a site are the tip. However, as you move down, the real work surfaces, such as interpreting testing statistics and actually making real changes.

As mentioned earlier, it’s a good idea to conduct A/B testing on a regular basis. In fact, it’s a cyclical process — you redesign, test, redesign, and test again.

There are a number of steps involved in A/B testing. These include:

• Figuring out what to test. If you’re unsure of what to test, take a look at some of your Google Analytics data to determine which pages have high bounce rates and high volume visitors. These are some of the easiest areas to target first. From there, ask yourself what you will be changing, and where. Are you adding a new CTA, changing a headline or swapping images around?
• Setting your goal. What do you want to achieve from this test? This could be a boost in newsletter sign-ups, more contact forms filled in or more ebooks downloaded.
• Coming up with a hypothesis. What do you think will happen at the conclusion of your test?
• Making your changes and creating variations. Ensure you note down the changes you make for each variant. There are a number of online A/B testing tools that can help you during this part of the process.
• Conducting your test. When will your test be held, and how long will you run it? There’s no hard and fast rule for the amount a time a test should run for. In fact, it is a heavily debated topic. You need to reach statistical confidence in order to capture the right data. Your results can become skewed if you hold your test for too long or too short of a period. For example, you might hold your test open for a day and find you have a 50% conversion rate. However, once you look closely at the data, you might have only had two visitors to your tested page on that particular day. There are online test duration calculators to help you with this.
• Interpreting results. Once your test has concluded, it’s time to analyze your results. If your variation is a clear winner, go ahead and switch to that version. If not, it’s time to start the process all over again.

Remember, in an A/B test you only test one new thing at a time in order to find out which option is more effective for your users. For example, here at Optimal Workshop, we tested our sign-up page. One option was a pretty plain version with a sign-up form, while the variant displayed logos of some of our many reputable clients. The banner was the only difference between the two variants. The testing was held for just over two weeks and at the conclusion, we reached statistical significance on the variant page in the form of a huge spike in conversions. In this instance, our conversions were the number of people who sign up for a free account.

Once we saw which version gave us the higher percentage of conversions, it was a no-brainer as to which sign-up page design to use.

Carrying out your A/B test and interpreting the results can be tricky. Fortunately, there are many great online A/B testing tools for you to use.

Up until this point, you’d probably want to use some sort of purposefully designed online A/B testing tool, such as Optimizely or Visual Website Optimizer.

Using Treejack for your A/B test

Optimal Workshop’s tools aren’t specifically designed for A/B testing, however you can use them for this purpose.

There’s only so much you can do by changing colours and visual features of your site. If you want to understand whether the categorization of your content can impact people’s success on your site then you can’t easily do this with A/B testing because it would require you to change your entire site. This is where our very own tree testing tool, Treejack, comes in handy, because you can mock up different structures and test them before you rewrite all your content.

Let’s say you wanted to test a new content structure for your site. Using Treejack, you can create two separate tests containing the different versions of your content structure.

What’s really important here is that you include the same tasks for each test. This is what’s going to help you figure out which variant performed better.

Once your tests are created, it’s time to recruit participants, whether you do this yourself or through Optimal Workshop’s recruitment service. Remember, as you’re testing two different content structures, you can’t send your tests out to the same pool of participants — that may skew your data. In order to reach statistical significance, you need to keep the tests separate.

Using Treejack to conduct your A/B test means you won’t have to rely on significance calculators or dabble about in your site’s coding; the test will contain your proposed content structure and it’s all performed in a certain time frame by a certain group of people. Easy! As a cheeky little extra, we’ve written up a short post discussing a way to change the javascript code on your site to direct participants to different tests, if you want to take that route instead.

At the conclusion of your test, you’ll have a few pietrees up your sleeve to help you figure out which variant was the most successful. Take a look at each task’s pietree and compare the results between the two tests. Each pietree should tell you the route your participants chose when completing your tasks. If they’re getting quite lost, that particular version of your content structure probably isn’t ideal.

If the results from both of your tests aren’t showing great results, it’s time to start the process again!

Differences between A/B testing and multivariate testing

If you want to test out a number of different features, in different combinations, all on one page, you’ve stumbled into multivariate testing territory.

It’s in a very similar vein to A/B testing, except while A/B testing is limited to just one change or two versions of an item, multivariate allows a number of changes. This allows you to test your hypothesis with multiple variables.

Let’s revisit our earlier example of testing a content structure. In a multivariate testing scenario, you’d create more than two Treejack tests containing the different versions of your content structure. The same rules for participants apply here — you need to ensure each test has its own set of participants in order to get reliable data. It can sometimes be tricky to get the right number of participants and this is where Optimal Workshop’s recruitment service can be really valuable.

If your demographics are really niche and you can’t find an adequate number of participants, you have two options: hold your tests with the same group(s) but with a couple days between tests, or use A/B testing instead.