Actionable Data

We have added an experiment duration calculator to our Analytics Buddy chrome extension that is based on the Google Website Optimizer calculator. When you are considering setting up an AB experiment you’ll want to know how long it’s going to take before you get a valid result. If it takes too long to get an answer it may not be worth implementing it.

Here is what I have learned so far.

For practical reasons it often doesn’t make sense to use the e-commerce conversion rate. The experiment duration will be too long even for busy sites.

But that’s OK since there are fundamental reasons not to use e-commerce conversion rate. In addition to limits on your time and resources, you can only take action on factors that are under your control. I would argue that the decision to buy involves many factors that are out of the control of the experimenter:

• Price. If your price is not competitive in the marketplace, people will not buy.
• Shipping costs. If your shipping costs are too high, people will not buy.
• Poor experience with call to 800 number.
• General brand trust and recognition built up over time.
• Various random and unpredictable events happening.

You also don’t want to “wait for significance” and run an open-ended experiment. Give an experiment a couple of weeks to run and then move on. I believe it’s better to run and learn from 20 quick experiments that focus on optimizing mini goals than to have one experiment based on e-commerce conversion rate. By winning these individual battles, you’ll win the war.

What I think we can do is focus on the immediate call to action on the test page. If you are sending traffic to a section page, how many people make it to the item page? If you are testing an item page, how many people add to cart?

You need to track these mini goals in web analytics:

• Non-bounce rate.
• Item page views. This is now easily done with non-interaction events in Google Analytics if you can’t segment by page name or title.
• Add to cart.

If you want to play with some data yourself, give the Analytics Buddy extension a try. The calculator is accessible on Landing Page reports and uses goal conversion rates for the selected row. You’ll need Chrome and be using the new version of Google Analytics.

Too Long:

Using Item Page conversion rate. Just right:

Time is the most precious resource, so hopefully this tool will make it easier to find sensible landing page experiments!