A / B and Multivariate Testing: Get True Results (Part 1)

Added on December 16, 2019 - Category: Statistics
A/B testing

Learning how to perform A/B and Multivariate Testing is of paramount importance for any webmaster. Here we’ll see what these techniques are, and how to use them.

A true user experience designer skill set includes the ability to test their premises with users and/or stakeholders. Before, during and after the launch of a product, service, website, etc.

For this purpose, there are countless techniques, tools, approaches, methods, etc. 

Although sometimes there are different ways of measuring something, there is usually ONE way that is the most appropriate. It could be by its methodology, or by the clarity of the results obtained.

Or it could be defined by the available possibilities (budget, audience reach, lack of baseline data, etc)

Availability example: suppose I design a User Experience for users located in France and I am in Japan. It will be very difficult for me to use guerrilla testing techniques such as Hallmark Testing. Even if I know that this technique is the most suitable for my needs.

However, if I have the financial means, I can hire someone in France to do it. Or replace this test with another to which I have access.

A / B Testing and Multivariate Testing

Two examples of the most common tests we perform in User Experience and HCI, especially when dealing with web pages or apps, are:

  • A/B Testing
  • Multivariate Test

Both tests are easy to perform at a very low cost and with excellent information returns. First, because of their simplicity.

Second, because both basically compare one element with another.

Because of this, it is very common for both methodologies to be confused. Specifically in the sense of when to use one or the other method. 

Let’s keep in mind:

A/B Testing and Multivariate Testing are not the same and may return conflicting results

A / B test and when to use it

An A/B test is basically a comparison of results between two versions with minimal differences. To be clear: only one difference.

Examples of these differences are: colors, slogans, CTA (Call to Action), typography, layout, ui design, etc. However, to perform the test correctly, only one parameter must be modified at a time.

A / B testing and multivariate testing: Example of A / B

A / B testing is widely used in landing pages and email marketing because they are very dynamic types of communication actions. Also, they can be modified very quickly, allowing almost immediate multiple comparisons.

The procedure to perform this test is as simple as comparing the results of two almost identical versions except in one detail. From this comparison, we can define what worked best based on the parameter that best suits our needs, such as CTR (click-through ratio) or total sales. For example, if version A has a CTR of 15% and version B has a CTR of 25%, version B is the best for our purposes. 

For the purpose of these measurements, there are tools integrated to specialized software. But depending on our needs, we can take the data from our server, or simply use Google Analytics.

DOs and DON’Ts of A / B Testing (what to do and what NOT to do)

What to do

  • We must always measure a single variable
  • We should always try to make the contexts as similar as possible. Pretending to compare a sales action at the beginning of the month with another at the end of the month will obviously give very different (and perhaps false) results.
  • We must always be clear about what to measure with an A / B test. There’s no multi-page or site-wide A/B testing.
  • It is a good idea to have an automated A/B version generation system, which can be easily generated with email services such as Mailchimp, or even creating special templates with WordPress.
  • The text and slogans are usually of fundamental importance. Not everything is buttons and colours.
  • A well-performed A/B test can generate revenue increases of up to 3000%. Obviously, a poorly performed test will cause us to lose that possibility and may even cause us to lose money in a poorly conducted campaign.
  • Let’s always stick to the data, no matter if we like it or not.

What NOT to do

  • Never measure more than one item at a time.
  • We never believe that a version is final. Once we find a winning version, we must test it in a variety of contexts.
  • It is preferable to avoid long and complex versions in favour of simple and smaller versions.
  • We must avoid taking the data as one-dimensional. The surface data obtained in an A / B test can hide another subset of data that contradicts our first impression.

Never trust that our instincts are going to be correct: the purpose of these tests is to confirm or deny the hypotheses.

Ready to continue with the second part: multivariate tests ?

Disclaimer: This content was translated to English from the original we wrote in Spanish, available in UXpañol

We can improve your business!

Let us help you with the best solutions for your business.

It only takes one step, you're one click away from getting guaranteed results!

I want to improve my business NOW!