[Update 03/05/2012: Added support for the new Google Analytics UI]
[Update 11/03/2011: We have since added goal conversion rates as well. However, please note that it uses the overall goal conversion rate, which is the sum of all goals. Use appropriately]
Can you be sure that a conversion rate of 5.56% is better than 4.87% if you only have a few data points? It’s important to think statistically when working with this kind of data.
I wanted to share a Z-Test calculator that you can use directly in Google Analytics via a simple bookmarklet. It uses the jstat library. You’ll need to use the new version of Google Analytics and have e-commerce tracking enabled as it works on the e-commerce conversion rate.
Please drag the button below to your bookmarks bar (you’ll need to be viewing the post on my site directly, not in an RSS reader):
And then click on it in any report that has Plot Rows checkboxes and select the e-commerce tab. Or take a look at the 2 minute video:
(You may want to watch a larger version directly on youtube)
A couple of notes. I decided to show the confidence interval so that you can decide if you want to use a 95% confidence level. Hat tip to my friend Kevin Potcner for giving me feedback on this.
Doing a statistical analysis on observational data will perhaps not give you quite as strong a signal as from a controlled experiment, but it’s still a valid approach. Try to isolate the question you are trying to answer by segmenting your data.
Have fun!
Michael
P.S. I tested it in Chrome and FF on a Mac, but if it doesn’t work somewhere else I am sorry. It’s just a tool we built for ourselves, which we are now sharing in the hope that it may be useful for other folks. Feedback is of course appreciated!
Follow me on 
{ 7 trackbacks }
{ 19 comments… read them below or add one }
Great tool. I will be using this on some of my clients. Any chance you will be adjusting this to work on all types of conversions?
This is really useful, thank you.
I have one suggestion, please. I wonder if it would be possible to change the wording so that it included a reference to the fact that you’re testing the significance of the difference in e-commerce conversion rate.
The use case I have in mind is where the button is being used by someone other than the person who installed it. In such a case they might guess that some other data is being tested, such as revenue or transactions.
Great suggestion Tim – added the text. You may have to clear your cache to get the newer version
I knew someone would ask that question! The challenge is that I don’t know how many goals you have set up, and having the user pick the desired goal might get a bit tedious. Unless you think it makes sense to use the overall goal conversion rate? See here
Michael: There are many reasons to love you, this is one more. : )
Thanks for creating this bookmarklet. We were just in a meeting and someone was emphatically arguing moving more marketing $$ into a strategy that was converting at 1.32% compared to the other one that was at 0.78%. I said…. wait a min… let me check something…. “63% confidence the rates are different”. Ok let’s wait a little bit!
Again, thanks so very much.
-Avinash.
Gosh Michael, that was quick. I actually spotted the change when I was using the bookmarklet and headed over here to say “thank you”.
Re the ‘overall conversion rate’ question:
Please don’t do that as a default. In fact I wish it was possible to hide that pesky metric in most ecommerce profiles. It certainly seems to confuse my clients when they have a whole series of micro-conversion goals.
Tim
Thanks for the kind words, Avinash! We have added the calculation on the overall goal conversion rate with a little disclaimer.
Hi Michael,
Just a quick comment, I am not certain what population of Analytics users are still under the old version. However, being the fact that this bookmarklet is great, you may want to state something in the nature that this tool only works in the new version of Google Analytics. It took me a few tries before I realized you were using the new version. When I switched over to the newer version of Analytics, your tool did what you show in your video. Again, I am not sure if my comment will even be relevant or important, but just thought I’d put in my two cents.
Great Tool though..
Emilio
Nice little tool. Really shows what you can do with jStat! Way to go, Michael! keep on developing!
Thanks Greg – any other ideas – please let me know!
I’m sure you’re working on it, but making that button work EVERYwhere would be the awesome next step!! So, go, Michael, Go!
just my feedback:
installing the button worked properly, error messages poping up, if the sample size is to small works – but no result if the sample size is big enough. Clicking on the button then just results in – nothing.
While I love the idea of easily calculating statistical significance for my GA results, I’m experiencing the same troubles as joohannes (see post above me). Any ideas on what the problem might be?
I´m having the same issues as joohannes and Yoram above. Any solution to this? Using Chrome.
Hi everyone who has tried to use the bookmarklet, but was unable to get it to work. Can you please send me send me a screenshot or screencast and I would be happy to take a look.
Michael figured out what I was doing wrong. The bookmarklet will not work when you compare date ranges. As soon as I had just one date period, it worked great.
Is there a way to create a z-test (with various levels of confidence @ 90/95/99 etc.) as an Advanced Segment, Michael? That would actually provide an interesting visual in terms of almost a ‘cartesian layout’ capability. I know that’s really vague, but I wanted to throw it in as a brainstorming element.
The bookmarklet’s fantastic! Any chance you can create one for Adwords? It’d be incredibly useful for comparing ad variations.
Hey, I’m getting an error box which says “please select TWO rows to compare” Any one else having this issue. I’m using chrome and within ecom tab.