Part 1 of 2. A/B testing done right, one variable, similar contexts, and always trust the data, even when it proves your favorite design wrong.
An A/B test compares two versions that differ in exactly one element (a color, slogan, CTA, layout) and measures which performs better on a chosen metric like CTR or sales. It’s cheap, fast and ideal for landing pages and email. The cardinal rules: change only one variable at a time, keep contexts as similar as possible, and always follow the data over instinct. A/B testing is not the same as multivariate testing (MVT), which changes several variables at once (covered in Part 2).
Key takeaways
- A/B testing compares two near-identical versions differing in exactly one variable.
- Pick a clear success metric (CTR, sales) before testing; there's no site-wide A/B test.
- Keep contexts comparable; testing start-of-month vs end-of-month gives false results.
- A/B and multivariate testing are different methods and can give conflicting results.
- Hypotheses often fail, and that's fine: the goal is to find what actually works, not to be right.
What A/B testing is (and when to use it)
An A/B (split) test compares two versions with a single difference, then picks the winner on a metric like click-through rate or total sales (if A gets 15% CTR and B gets 25%, B wins). It’s widely used on landing pages and email because those are dynamic and quick to change with a little HTML/CSS or a plugin. Measurement can come from specialized tools, your server logs, or analytics. A key constraint: choose in advance the one thing you’re measuring.
DOs and DON'Ts
| Metric | Value | Change | What it means |
|---|---|---|---|
| Measure one variable | DO | Change a single element per test | Never test more than one item at once, or you can’t attribute the result. |
| Keep contexts similar | DO | Compare like with like | Same timing, audience and conditions; otherwise results may be false. |
| Follow the data | DO | Stick to results even if you dislike them | The test’s job is to confirm or deny a hypothesis, not flatter instinct. |
| Treat a winner as final | DON'T | Re-test winners in new contexts | No version is final; validate it across situations. |
| Read data one-dimensionally | DON'T | Look beneath the surface metric | Surface data can hide a contradicting subset. |
| Build long, complex versions | DON'T | Prefer simple, small variants | Simpler versions isolate the effect and are easier to compare. |
The team claims a well-run A/B test can lift revenue dramatically (they cite up to 3000%), while a poorly run one loses that upside or worse. Text and slogans matter as much as buttons and colors, so don’t test only visuals.
Real example: when the 'wrong' version won
In a 2016 landing-page test for a client (HealthyWage), the hypothesis was that Version B (showing the exact prize amount) would beat Version A. It didn’t: Version A won by 13.8%. Why? User psychology. Not knowing the exact amount was more attractive, because people expected that if more players joined, the prize would grow, a pattern known as optimism bias (related to wishful thinking and the valence effect). A failed hypothesis means nothing on its own; what matters is discovering which design actually works. Continue to Part 2 on multivariate testing.
Comparing two versions of a page or element that differ in exactly one variable, to see which performs better on a chosen metric like CTR or sales.
A/B testing changes one variable at a time; multivariate testing (MVT) changes several at once. They are different methods and can give conflicting results.
Changing more than one variable at once, or comparing versions in dissimilar contexts (like different times of month), which produces misleading results.
Decide in advance: usually click-through rate or total sales/conversions, matched to your actual goal. There is no valid site-wide A/B test.
That’s normal and useful. The point isn’t to be right but to find, via scientific method, which version genuinely performs, as in the example where the ‘wrong’ version won.
Want tests that actually move revenue? Talk to Dorve.
Start a project