A/B Testing and Lift

Great article by Kevin Hillstrom about A/B testing for email campaigns. In a nutshell, it’s not about open/click/conversion metrics but about incremental lift. You calculate lift by splitting your list into two groups: one group gets the mailing and the other one doesn’t. Then you track revenue from both groups and work out the difference in revenue per customer.

What I wanted to add is that this logic also applies A/B testing on your website. If you are thinking of trying out a new creative/offer on your site, just make sure to test it first by doing A/B testing.

Just watch out if your testing tool only gives you a conversion rate. You may have to get revenue data in your web analytics.

In particular, if you are offering coupons or incentives to visitors, which of course lowers your margin, you need to take into account incremental revenue and not (just) conversions. As Kevin puts it, you don’t want “opens/clicks/conversion to take credit for orders that would have happened anyway”. Well said.

Maybe offering a coupon does increase conversions, but what about your profits?