Website Optimizer and custom segmentation
In addition to that Shawn makes a great point about the fact that picking a winner in a test is purely based on one metric: conversions. Perhaps this is not “deep” enough. After all, you could just set up a test headline that promises “50% off of all items” and you’d surely see conversions go up. But profitability would go down too. Apparently Shawn has something in the works about showing experiment data right in Google Analytics – that would be very cool.
One approach I was working on was with Dan Thies who asked me about the possibility of tying in a Website Optimizer experiment with Google Analytics’ custom segmentation capabilities via the setting of the _utmSetVar variable. Essentially _utmSetVar allows you to tag site visitors with attributes that you choose, e.g. “Newsletter signup”. You can then segment and view reports just for people who have been tagged as a “Newsletter signup”. Justin Cutroni also has a great article about custom segmentation.
Anyway, what Dan and I have done is set a custom segment for each Website Optimizer variation. To put it more concretely, imagine a simple test: Orginal and two variations: Deal1 and Deal2. So some visitors will see the original page and some will see one of those variations. Website Optimizer will count the impressions and conversions (and eventually make recommendations). However at the same time, we are setting _utmSetVar depending on what the visitors gets shown, so the Original will be shown in Google Analytics as the Original custom segment, Deal1 in Website Optimizer will be counted as Deal1 in Google Analytics, and so on.
Well, this approach seems to work! See below the screenshots of an actual experiment:
(click image for larger) This is the report section of the Website Optimizer test. Original = Original, Combination 1 = deal1 and Combination 2 = deal2
In Google Analytics, this is what the user-defined report looks like:
So, you can see that that those combinations are now seen in Google Analytics, including actual $ values. And you can cross-reference those custom segments in other reports too.
So, on the face of it, this is one way to tie Website Optimizer data into Google Analytics.
The only question I want to toss out though is how to interpret this data and I invite your comments. Website Optimizer works on a session basis or a 30 minute visit; if someone is inactive for 30 minutes then the visitor may be shown a new experiment variation and Google Analytics would count this as a new visit. However, the cookie associated with _utmSetVar lives for a very long time. The question I have is whether this matters or not because _utmSetVar would always be reset if a new experiment variation is shown. So it should be OK, but I am not entirely sure.