E-Mail Marketing Metrics and Bronto

This entry was posted on Tuesday, July 6th, 2010 at 2:30 pm and is filed under A/B Testing, Email Marketing, Mike's Opinions, Web Design, online marketing.

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

I do a lot of email marketing, deploying 3-5 marketing email campaigns a week across various brands. When I first started doing the email marketing at my current job, we were pretty much only doing subject line tests with each deployment. However, we’ve recently gone a step further and started doing full content testing, which I think is a great thing. The question that comes up though is what is the marketing metric that determines success?

We use Bronto email marketing software for all our in-house e-mail marketing campaigns. Bronto is definitely one of the better e-mail marketing software platforms out there. It’s user-friendly, offers great support, and includes great metrics for all of our campaigns. It’s also very easy to set up and deploy email tests based on factors such as differences in subject line, sender, date/time of send or even full message content. And as mentioned, all the metrics to go along with the test.

We recently ran a full content test using two different creative variations promoting the same product. The basic hypothesis of the test was this: “Does having the product shot in the top section of the email help with conversions?” To test this hypothesis, we created 2 different email creatives, one had a nice visual of the product shot (a case of red wines), along with the initial UVP in the header section of the email (above the fold), the other used a strong visual of the product in use (a glass of red wine), with the same messaging and UVP.

The body copy for both emails was almost identical, however the layout of the body copy changed slightly, and on the email with the product in use, we added the product shot to the bottom of the email (a better test would be to not include the product shot at all, but I’ll save that for another test). You can view each version by clicking on the images below:

I should say that typically, we always include the product shot in the header of the image. We’ve seen pretty good success with this, so it’s always hard to convince the hippos (higher paid peoples opinons) to change, but I’ve been pushing hard to try using more lifestyle shots in the header to test the effect on coversions. And through Bronto it’s easy to test this and have the numbers to support (or disprove if the case may be) these “hunches”.

We decided to run a 50/50 split test to our master email list, and Bronto provided some great results of our test. Here is the results summary:

Opens: While Bronto allows for subject line tests, to keep a clean test, we used the same subject line for both, so open rates were as expected very close, 24.7% vs. 24.2%.

Click-throughs: The first metric that would help determine success, the CTR fell in favor of the version with the sole wine glass, 3.1% vs. 2.2%.

Clicks/opens: Obviously, with opens such a close number, and variance in clickthrough being somewhat significant, the clicks/opens rate will fall in the same favor as click-throughs, 12.4% vs 9.1%

Conversions: Here’s where things get tricky… the difference in conversions was only 3 orders. Again this was in favor of the version with the wine glass, but the difference was disappointingly small.

Conversion Click Rate: The numbers continue to shift now, with the version with the bottles beating out the wine glass 10.6% to 14.4%.

Conversion Revenue: With the conversions being so close, the overall revenue of course was very close as well (although it is important to note that purchases outside of the email promo product were counted in the test), with the wine glass slightly edging out the bottles.

Revenue per click: This is where a clear winner gets trickier again. The bottles version, while having less conversions, had not only a higher conversion rate but a 35% higher revenue per click (pretty significant considering the other numbers were so close).

The winner? Well, that’s a little hard to say…

If you recall, our hypothesis for this test plan was “”Does having the product shot in the top section of the email help with conversions?”. Since we noted “conversions” as our primary metric in our hypothesis, technically we should say the wine glass version is the winner as it had more conversions than the other. However, it’s conversion rate was about 30% below the bottles version, and with a 35% higher revenue per click the it’s hard to label the bottles version as a “loser”.

Strictly based on conversions and revenue, the wine glass version should be declared the winner. However, the real win here is that we can extract a lot of learning from this test, without a significant NEGATIVE impact to our weekly campaign. And it’s something that Bronto, our email platform of choice allows us to do with very little effort.

For some great tips for testing success from Bronto, you can read this article.