Part 2 of 2. Multivariate testing changes several elements at once to find the best combination, powerful, but hungry for data.
Multivariate testing (MVT) is a statistical method that tests several changed variables at once to find the combination of elements that performs best on your KPIs. Unlike A/B testing (one change, two versions), MVT compares multiple versions with multiple changes. There are four main methods: discrete choice, full factorial, optimal design, and robust design (Taguchi). Its big limitation: as a factor-analysis method it needs a lot of data, and adding variables makes it geometrically harder to tell which element drives the result, so keep to 2-3 versions at a time. Note: this is not MANOVA (multivariate analysis of variance).
Key takeaways
- MVT changes several variables at once to find the best-performing combination, judged by KPIs.
- It's a direct data-comparison test, like A/B, but with multiple changes across multiple versions.
- Four methods: discrete choice, full factorial, optimal design, and Taguchi (robust design).
- MVT needs large data volumes; combined elements can behave differently than they do alone.
- Complexity grows geometrically, so limit tests to 2-3 versions and use heatmaps/scroll data to clarify.
The four MVT methods
| Method | How it works | Best for |
|---|---|---|
| Discrete choice (choice modeling) | Makes changes at the exact moment and context of purchase. | Big e-commerce (Amazon, eBay) |
| Full factorial | Serves each version to an equal share of users, making the statistics straightforward and accessible even without deep stats knowledge. | Most common / simplest |
| Optimal design | Tests many UI versions in ‘waves’ (as often seen on Facebook); needs lots of user data to build sub-sets. | High-traffic sites |
| Robust design (Taguchi) | Taguchi orthogonal arrays reduce the number of combinations while keeping data reliable. | Expert statisticians |
Considerations and limits
MVT is powerful but data-hungry: as a factor-analysis method it needs a large sample for reliable results. Adding complexity makes it harder to isolate which element helps or hurts, and elements that test well separately can behave differently in combination. Clean up the picture with complementary tools, scripts on tested elements, scroll analysis, heatmaps and overlays. Because difficulty grows geometrically with each sample, use no more than 2-3 versions at a time. For the fundamentals and DOs/DON’Ts, see Part 1; for the underlying design, see factorial design.
A statistical method that changes several variables at once across multiple versions to find the combination that performs best on your KPIs.
A/B testing changes one variable across two versions; MVT changes multiple variables across multiple versions to optimize a combination.
Discrete choice, full factorial (most common), optimal design (wave testing), and robust design / Taguchi orthogonal arrays.
It needs a lot of data, and combining many variables makes it hard to isolate which element drives the result, so 2-3 versions at a time is recommended.
No. MVT here is a conversion-optimization testing method, not the multivariate analysis of variance (MANOVA), which is a more complex statistical test.
Need multivariate optimization done rigorously? Talk to Dorve.
Start a project