Using A/B Testing, Factorial Design, and Multivariate Tests for Deep Visitor Insights

Data is the driving force of website design. Changes should never be made on a whim; every change requires an investment of time and resources, and website owners shouldn’t consider making changes before ensuring that they’re justified. This applies to both small site owners making their way in the world and bigger agencies that need to defend their marketing spend to stakeholders.

Of course, different types of testing structures provide different information on website performance. Below, we’ll review a few of the most common options that marketers use.

Start With A/B Testing

Most of us are familiar with A/B testing. It’s one of the most basic forms of market research, either asking subjects to compare their impressions of two options or routing web traffic to two different options and measuring the differences in behavior between each. This can be a single change (such as a CTA button colored blue or green) or a group of changes (such as two completely different website layouts).

For example, you might be considering four different changes to your homepage:

1. Changing placement of your CTA
2. Changing the style of your navigation bar
3. New headline copy
4. Adding customer testimonials

With A/B testing, marketers typically assess each option individually. They use their established website as a control and then change elements one by one to determine what impact each has on conversion rates. This splits traffic into one of two buckets: The control (A) and the change (B).

This type of research is a great start for judging the effectiveness of your web design, but it offers only one perspective. You’ll need deeper analysis to go further.

Adding Multivariate Testing (MVT)

MVT takes A/B testing a step further by isolating the impact of individual changes and comparing them against each other in every possible combination. This process provides great insight into the relationship between different elements. Going back to our above example, let’s say that our A/B test gave us the following data on these changes:

1. CTA placed below the fold (+10% conversions)
2. Navigation bar turned into drop down menu (+5% conversions)
3. New headline copy (+10% conversions)
4. Removed testimonial from homepage (-5% conversions)

If you compared all of these together under a single A/B testing group model, you’d get a net increase of 20% conversions. Great! Your changes worked…right?

Not necessarily. When looking at the data individually, it’s clear that this isn’t the optimal outcome. You need a way to separate the negative changes—number four in our example—from the positive. The problem is, this is easier said than done with basic A/B testing. This is where MVT comes in.

The MVT model lets you test combinations of variables in tandem to determine which combination produces the most overall benefit. With the four variables noted above, a proper MVT would produce 15 different combinations of results. It’s functionally the same as a series of lengthy A/B tests but done in a shorter time. So, why would anybody bother with A/B testing if MVT is so much more effective?

Tests Need Traffic

The biggest limiting factor for most websites is traffic. Tests like A/B and MVT depend on statistical significance, a term indicating that the experiment has been run enough times to produce valid and repeatable results. Without statistical significance, the results of a test could be considered mere chance, or luck.

This is the key drawback of MVT testing. To perform a proper A/B test, you’re comparing two variables: A vs. B. Your website traffic is split in half, routed between these two options to determine which one is better.

With MVT testing, there are many more options. Even assessing three variables at once (A, B, and C) requires traffic to be split into eight different buckets:

And as more variables are added, this traffic division increases exponentially. Unless your website has substantial traffic to spare, MVT makes it tough to come up with statistically significant data.

This brings us to our third testing method: Factorial design.

One Step Further: Factorial Design

Similar to MVT, factorial design involves testing multiple variables at once. The difference is that you don’t have to test every combination of variables—you’ll only test specific combinations that you think will produce the biggest improvements to your website.

Here’s an example. Let’s assume you have a small website that receives 100 visitors each month, with a total conversion rate of one percent. This is our baseline, or control:

From this, let’s apply change one from our above example.

Simple enough. This is the same as a basic A/B test, producing a 10 percentage point increase in conversions for a total conversion rate of 11%. (The formula used to calculate change in conversion rate here is (11 / 1 – 1 = 100%)

From here, MVT would have us perform this same test for every variable (and combination of variable) listed above. As we’ve already discussed, this is time-consuming and diverts too much traffic between each option to make the results very useful.

Thus, we have factorial design. Here’s how it works. We’ll assess change two from our above list, except instead of comparing it against our control, we’ll apply it to our change one results, which becomes our new control:

This gets somewhat complex, particularly as you layer on these changes. But the key takeaway for factorial designs is that you’re performing all of the same calculations as MVT, but with fewer steps. Rather than testing each change individually, you’re grouping them together to receive the same overall data without drastically dividing your web traffic.  

Of course, statisticians will note that our above framework includes some assumptions that don’t hold true 100 percent of the time, but overall, the formula is reliable enough to make factorial design an effective middle-ground between basic A/B testing and extensive MVT calculations.

Combining Methodologies for Audience Analysis

There’s a lot more to these different types of analyses, and we’re only scratching the surface. But we hope this rundown gave you some idea of the possibilities at your disposal. As always, companies will need to determine which methodologies are most feasible with their limited resources—and which will produce the most valuable insights for their website optimization.