Article about A/B and Multivariate Testing. If you want to start with part 1, please go to A/B and Multivariate Testing. What are they? (Part 1)
First of all, and to avoid confusion, this test is not the Multivariate analysis of variance, which is a much more complex test. This one can be used in subsequent steps to obtain statistical data of large information. However, it is not the Multivariate Test we refer to here.
Definition and Application
The multivariate test allows us to corroborate a hypothesis with multiple possible variables.
But A/B compares 2 versions with minimal differences. While multivariate compares multiple versions with different changes. You can use the insights from these tests to fine tune until you get the best performance.
To perform this test, we will take two or more samples * with different elements and analyze their performance based on KPI (key performance indicators, or KPIs ) that we need to analyze. Once we have the data we compare and see which is the best version.
As you can see, this method is very powerful, but it has obvious disadvantages. The main one is that being a method of factor analysis, it requires a large amount of data to obtain reliable results.
Another disadvantage is that when we add complexity to the samples, it could be difficult to identify the elements that generate a positive or negative difference.
In addition, certain elements together can generate very different results than the same elements separately.
One way to “clear out” the data obtained to have an even more accurate view is by using complementary tools. For example adding scripts in the elements to be tested, the analysis of page scroll, heat maps, overlays, etc.
* Note: Although we can take multiple samples, they will generate a geometric increase in the difficulty of analysis. Thus, it is recommended to use no more than 2 or 3 versions at a time.
Cover image: Steve Juvertson
Disclaimer: This content was translated to English from the original we wrote in Spanish, available in UXpañol
We can improve your business!
Let us help you with the best solutions for your business.
It only takes one step, you're one click away from getting guaranteed results!I want to improve my business NOW!