I’m not a big fan of subject line tests for a couple reasons. They’re overused and many don’t have a good chance of success because they lack a sound hypothesis.

If I undertake a subject line test with a client I’m going to do it strategically, to give us our best chance at a significant lift in bottom-line performance.

That’s what we did for an email that was part of a client’s 2020 holiday campaign. Here’s the full story, with tips for leveraging these learnings to boost your own bottom-line performance.

Where We Started

As always, we start with what’s worked in the past. This email was one of the top performers in terms of revenue-per-email (RPE) in 2019, so we decided to use it as our control. The products were updated for the 2020 season, but the subject line and preheader text did not need to be touched. Here they are:

You can only guarantee that the first 25 characters of each will be seen, so highlighting that ‘prime real estate’ we get:

Not bad. The fact that these are new, that they are holiday gift sets, and that you can save time and money are all in there.

But I had two goals:

• Increase performance on this email
• Create a formula we could use on future emails

The second goal allows us to increase the return on the test exponentially, since we can apply the learnings in an ongoing manner.

Where We Went

Using these formulas, here are the control and test elements, side-by-side, with the first 25 characters highlighted:

Not a huge difference you might say. But a good bit of thinking when into these. For instance…

• I wanted to make sure ‘value packs’ got equal billing with ‘holiday gift sets.’ ‘Value’ is a powerful word – not quite as powerful as free, but powerful. It reads like ‘discount.’  And it’s very different from ‘holiday gift sets’ which suggest multiple items in a single package – but not a cost-savings.
• I wanted to get replace ‘save time & money’ with something that had more of a holiday feel. I don’t think in those terms about holiday shopping. After a lot of trial and error, we came up with ‘Spend less, give more’ which supported the use of ‘value’ in the subject line as well as the ‘sets/packs’ idea of multiple gifts.

Small changes. Thinking it through took some time, but once that was done the cost to create the test version was minimal, since nothing else about the control changed.

How It Did

Pretty well.

As you know from the headline, our revenue-per-thousand-emails sent (RPME) increased by 43%. The control generated an RPME of just under $50 ($49.59) while the test generated more than $71 ($71.06 to be exact) for each thousand emails sent.

Note: We use RPME (revenue-per-thousand-emails-sent) here because it’s easier to spot the variance than it would be if we used RPE (revenue-per-email-sent). The variance remains the same; it’s just easier to see with the larger numbers.

Note: The sample sizes were very large – roughly 100,000 per cell (200,000 total send quantity) – and we did confirm that the variance was outside the margin of error.

Looking at the other metrics here we can get a read on what caused the test creative to win.

Let’s start by looking at the conversion-rate-from-emails-sent (CR from sent). It’s 0.012% for the control, and 0.020% for the test. That’s a lift of 66% for the test. This tells us that the test subject-line/preheader-text pair (SL/PH pair) drove more people to purchase. That had an impact on the results.

Looking at the average-order-value (AOV), we see that although more people purchased from the test version, those people spent less money on average than those who bought from the control. The test AOV is just under $355, while the control AOV is just over $416. That’s a variance of -15% for the test.

So, we have a higher CR from sent but a lower AOV for the winning test version.

Let’s look at the diagnostic metrics.

Here we see that the open rates are close, but the variance is statistically significant. The test lagged the control by 3%; it drove fewer opens.

The same is true of the click-through rate (CTR); the test lagged the control here by 13%. The test also lagged the control in click-to-open-rate (CTOR). Here the variance was -10%; the clicks were more behind the control than the opens.

This is yet another example where using a diagnostic metric, like open rate or CTR, as a key performance indicator (KPI) for the test would have been a bad idea. Because even though the test lagged the control on these diagnostic metrics, it bested the control on the only metric, the business metric, that really matters: RPME.

Your Next Step

Give this a try with your next email campaign and let me know how it goes!

Too busy to do-it-yourself? I’d love to speak with you about a consulting engagement! Reach out and let’s set a time to chat.