Multivariate Testing and A/B Testing. What are they? (Part 2)

Added on December 16, 2019 - Category: Statistics, Marketing, Theory

Article about A/B and Multivariate Testing. If you want to start with part 1, please go to A/B and Multivariate Testing. What are they? (Part 1)

Multivariate Testing

First of all, and to avoid confusion, this test is not the Multivariate analysis of variance, which is a much more complex test. This one can be used in subsequent steps to obtain statistical data of large information. However, it is not the Multivariate Test we refer to here.

Definition and Application

The multivariate test allows us to corroborate a hypothesis with multiple possible variables.

Same as the A/B Test, the multivariate test is a direct data comparison test. 

But A/B compares 2 versions with minimal differences. While multivariate compares multiple versions with different changes. You can use the insights from these tests to fine tune until you get the best performance.

To perform this test, we will take two or more samples * with different elements and analyze their performance based on KPI (key performance indicators, or KPIs ) that we need to analyze. Once we have the data we compare and see which is the best version.

Multivariate test.


As you can see, this method is very powerful, but it has obvious disadvantages. The main one is that being a method of factor analysis, it requires a large amount of data to obtain reliable results.

Another disadvantage is that when we add complexity to the samples, it could be difficult to identify the elements that generate a positive or negative difference.

In addition, certain elements together can generate very different results than the same elements separately.

One way to “clear out” the data obtained to have an even more accurate view is by using complementary tools. For example adding scripts in the elements to be tested, the analysis of page scrollheat mapsoverlays, etc.

* Note: Although we can take multiple samples, they will generate a geometric increase in the difficulty of analysis. Thus, it is recommended to use no more than 2 or 3 versions at a time.

Cover image: Steve Juvertson

Disclaimer: This content was translated to English from the original we wrote in Spanish, available in UXpañol

We can improve your business!

Let us help you with the best solutions for your business.

It only takes one step, you're one click away from getting guaranteed results!

I want to improve my business NOW!