Article about A/B and Multivariate Testing. If you want to start with part 1, please go to A/B and Multivariate Testing. What are they? (Part 1)
Multivariate Testing (MVT)
First of all, and to avoid confusion, this test is not the Multivariate analysis of variance, which is a much more complex test. This one can be used in subsequent steps to obtain statistical data of large information. However, it is not the Multivariate Test we refer to here.
Definition and Application
To easily define multivariate testing (also known as MVT), we can define it as follows:
Multivariate testing is an statistical method to test a hypothesis with several modified variables. These variables and tested and compared one to each other in order to extract a combination of elements that performs better than other.
Same as the A / B Test, the multivariate test is a direct data comparison test.
The difference it that A / B compares 2 versions with one (often minimal) change, while multivariate testing compares multiple versions with different changes.
You can use the insights from these tests to fine tune until you get the best performance.
To perform this test, we will take two or more samples * with different elements and analyze their performance based on KPI (key performance indicators, or KPIs ) that we need to analyze.
Once we have the data we compare and see which is the best version.
Multivariate Testing Methodology
As with any other statistical analysis method, we need to follow a strict methodology. This means we need to create experiments that are rigurously controlled following an already accepted framework or methodology.
When we work with MVT, we have four main methods to use:
Discrete Choice Method
Also called Choice modeling, this method consists in doing changes at the exact time of purchase, in the context of such purchase. This is a method used by the biggest eCommerce companies such as Amazon, eBay, etc.
Full Factorial Method
This is the most used, because of its simplicity. In this method, users are served with an equal amount of probabilities. This makes statistical calculation very straightforward and accesible to most people, even without statistical knowledge.
Optimal Design Method
Optimal Design is a very common method as well. It consists in a huge variety of UI design versions tested in “waves” (you might have seen this on Facebook). This method requires a lot of user data, therefore is used only by sites with lots of traffic that can create sub-sets of users.
Robust Design Method (aka Taguchi)
The robust design method (also known as Taguchi Methods or Taguchi Ortogonal Arrays) is a framework or set of methods that reduce the number of possibilities while still providing reliable data.
This method and its different methodologies are commonly used by expert statisticians and big data specialists since they require a lot of knowledge of statistics and maths.
Considerations
As you can see, this method is very powerful, but it has obvious disadvantages. The main one is that being a method of factor analysis, it requires a large amount of data to obtain reliable results.
Another disadvantage is that when we add complexity to the samples, it could be difficult to identify the elements that generate a positive or negative difference.
In addition, certain elements together can generate very different results than the same elements separately.
One way to “clear out” the data obtained to have an even more accurate view is by using complementary tools. For example adding scripts in the elements to be tested, the analysis of page scroll, heat maps, overlays, etc.
* Note: Although we can take multiple samples, they will generate a geometric increase in the difficulty of analysis. Thus, it is recommended to use no more than 2 or 3 versions at a time.
Cover image: Steve Juvertson
More Multivariate Testing Resources
Experiment!: Website conversion rate optimization with A/B and multivariate testing – C. McFarland
From web analytics to digital marketing optimization: Increasing the commercial value of digital analytics – D. Chaffey, M. Patron
Fabio is Bachelor in Marketing, Master in Human Computer Interaction and a UX Evangelist.
He worked for some of the biggest companies in the world. As such, he has developed entire systems and businesses for clients of all sizes.
Disclaimer: This content was translated to English from the original we wrote in Spanish, available in UXpañol
We can improve your business!
Let us help you with the best solutions for your business.
It only takes one step, you're one click away from getting guaranteed results!
I want to improve my business NOW!