A/B testing or Multivariate testing?

Web Analytics

Isn’t it cool to even have the choice to do A/B testing or Multivariate Testing courtesy of Google Website Optimizer?

I have been using both tools extensively over the past several months. I am surprised that many online merchants haven’t tried doing website optimization this way, although once they “get it”, they’ll want to run experiments all the time.

And why wouldn’t you? I find it very satisfying to let customers make the decision as to what works and what doesn’t, instead of the personal preferences a website owner or opinionated web designer. Don’t redesign your site just for the sake of it, or be told by someone that it’s time to get a “fresh look”. Some of the best converting sites are not pretty. At the very least, make sure you capture “before and after” data in your analytics solution.

A re-design by definition changes so many elements that you won’t know what specific elements contribute to your conversion rate. Is it the Hackersafe logo in the left nav or the Free Shipping message in the top nav? This is akin to doing an A/B test: you compare one page vs another and see what converts better. However, you won’t know what element contributes most.

For this reason, I prefer multivariate testing. Here, you essentially compare different elements on the same page. So you would test the Hackersafe element and Free Shipping element on the same page. The experiment will then show you what element contributes the most. You can even find out if a particular combination of elements converts better than another.

But there is another reason I prefer multivariate testing over AB testing, which I don’t think gets talked about much. Perhaps it’s not too huge an issue in practice, but as a heavy user of web analytics, I care a lot about getting good data all the time.

In an A/B test, if your A page is an important landing page, i.e. the first point of entry of your site visitors, consider this: the experiment will keep roughly 50% of visitors on A whereas the other 50% will get re-directed to the B page, as it should be. But, what happens to the referrer information? If the visitor sees page A the original referrer (e.g. Google organic) will be preserved. However, if the visitor lands on page B, analytics will think that the visitor came directly to the site. So, the original referrer information is lost.

Does this make sense? Anybody in the A/B camp disagree with me?