15 Surprising Email Subject Line A/B Test Results You Need to Know
Contents
How much thought do you give to the subject lines of your emails?
Are you A/B testing them on a segment of your subscribers before rolling the winner out to everyone, or using them as a way to improve and refine your subject lines each time you schedule in a new campaign?
If not, you should be. Your subject line is your digital calling card, and it needs to get that door opened. Nearly half – 47% – of recipients decide whether to open or discard based exclusively on the subject line, and 69% report an email as spam – the dreaded s-word – depending on their impression of it.
A/B tests are simple to execute (and even simpler if you use an email marketing platform with built-in split test capabilities). They’re also invaluable in helping marketers get more from their emails, and critical if you want to ensure you’re not only doing what works, but what works best.
“Never stop testing, and your advertising will never stop improving.” David Ogilvy
To get you started, and give you some ideas for subject lines you might want to try out yourself, here are 15 A/B tests that prove what works best isn’t always what you expect (which if you needed further proof, is one more great reason to test everything).
1. Including the word “free” in your subject line probably isn’t sending emails to spam (and can significantly increase open rates)
The Tumbleweed Tiny House Company used AWeber to split test two versions of a subject line – one containing the word “free,” and one without.
Subject line A: “It’s FREE. All the tiny houses on our site and more.”
Subject line B: “It features all the houses on our website plus more…”
The winner: Subject line A (26% more opens)
What’s going on?
The idea that using the word “free” in email subject lines can trigger spam filters is nothing new, and in some cases may be true (if you’ve got a history of sending spammy emails for instance, placing “free” in your subject line probably isn’t doing you any favors). The word frequently appears on every list of “words to avoid in your email marketing.”
That said, if you’ve got a clear track record with your email marketing platform, using “free” in your subject line is not only unlikely to affect the deliverability of your emails (because the sender’s reputation is now more important to where an email ends up than its contents) but as the test above shows, could significantly increase open rates.
“FREE! gives us such an emotional charge that we perceive what is being offered as immensely more valuable than it really is.” ~Dan Ariely, Professor of Psychology and Behavioral Economics
The reason why is simple – we really, really like free things.
2. And a synonym of “free” might increase open rates even more
A test using email marketing platform informz found that while the word “free” can boost open rates, using a synonym of it could increase them even further.
Subject line A: “Complimentary Online CME Courses”
Subject line B: “Free Online CME Courses”
The winner: Subject line A (3% more opens)
What’s going on?
There’s only a small discrepancy in open rates, which indicates that the same premise seen in test one must still be at play (people like free things). The word “free,” however, can be linked to “low value” or “low quality.”
“Complimentary” doesn’t share these same connotations, which allows you to tap into the consumer’s desire for a freebie, but without devaluing your product.
Try this free (see what I did there?) email subject line tester to compare open and click rates of various synonyms.
3. Personalizing your subject line could be as simple as adding a “you”
Wishpond used A/B testing to help increase the overall effectiveness of an email campaign they used to boost sign-ups to their VIP demo. They found that some very simple personalization – adding the word “you” to the subject line – could boost open rates.
Subject line A: “Social Media Stats You Need to Know for 2014”
Subject line B: “Social Media Stats for 2014”
The winner: Subject line A (11% more opens)
What’s going on?
Personalizing an email subject line is a powerful and effective way to increase open rates, but there is some evidence to suggest that consumers have become desensitized to the impact of seeing their name in their inbox. The same Wishpond study found a subject line with a first name performed slightly worse than the exact same one without the name added.
This alternative addresses the recipient directly and feels personalized, but without using the recipient’s name.
4. Money isn’t everything
A test by Optimizely set out to establish whether their audience was more motivated by discounts or by benefits.
Subject line A: “Get Optimizely Certified for 50% Off”
Subject line B: “Get Optimizely Certified & Advance Your Career”
The winner: Subject line B (13.3% more opens)
What’s going on?
Some audiences will be more motivated by benefit and self-interest than straight discounts (not to mention the over-reliance on discounts from spammers). Try this out for yourself by writing subject lines that touch on how your product or service can make a positive difference to their lives or businesses.
5. But context is everything
This is another test from Optimizely, from an email designed to promote an ebook they created. They made one small change to variant B – adding in the type of content they were promoting.
Subject line A: “Why You’re Crazy to Spend on SEM But Not A/B Testing”
Subject line B: “[Ebook] Why You’re Crazy to Spend on SEM But Not A/B Testing”
The winner: Subject line B (13.3% more opens)
What’s going on?
Ambiguous subject lines drive curiosity, creating a knowledge gap which can increase open rates. But there’s little doubt that they can decrease click-through rates and conversions, too. This is because a vague subject line lacks any targeting. Open rates might be high, but that’s because your recipients couldn’t resist finding out what’s inside the email.
It doesn’t mean they have any interest in what it contains.
The antithesis to an ambiguous subject line is one that details exactly what the email contains. Not only should this increase click-through rates, but as seen in the test above, it could increase open rates too (at least when compared to a similar, but slightly less descriptive version of the subject line).
6. Emails sent from real people instead of a company could see higher open rates and click-throughs
HubSpot conducted an A/B test to find out whether their audience responded better to emails that featured the company or a real person as the sender.
Subject line A featured “HubSpot” as the sender’s name
Subject line B featured a real person as the sender’s name
The winner: Subject line B (0.53% more opens and 0.23% more clicks – it also resulted in 131 more leads)
What’s going on?
We know an email that features a company as the sender’s name is part of an automated email campaign that’s probably trying to sell to us. Using a real person’s name feels more personal (even if it’s not).
7. Clarity trumps creativity
We’ve already seen evidence to indicate that subject lines which are upfront about the contents of an email are more effective than those that try to create “mystery” by covering it up, but this series of tests from AWeber proves the theory pretty much beyond all doubt.
They ran a simple A/B test across a range of emails. Each one pitted a “clear” subject line against something more creative.
Here’s a sample of some of the subject lines they tested.
The clear subject lines performed better than their creative counterparts across all metrics, and led to an average increase in responses of 541%.
Chris McCarron, owner of Conversion optimization agency GoGoChimp experienced a similar result when he used Mailshake to test a clear subject line; using the simple, straightforward subject line “Looking for quotes” resulted in a 51% open rate, 9% click-through rate, and 9% reply rate.
Chris details the results and thinking behind the subject line in his video on The Science of Cold Email Marketing.
[lyte id=”5v3hMggTdyo” /]
In the video, he explains that the primary success behind their cold email campaign rests in the simplified message they landed on by “pretending as if we’re explaining our offer to a 4-year-old”. He adds “We all live in a professional bubble, and this sometimes makes it hard to clearly express what it is we do, why someone needs it and the benefits that it brings”.
Austin Belcak, founder and CMO of Cultivated Culture, uses the subject “Mentioned You” when promoting his content. Most people get excited at the possibility of being featured and this subject plays right into that. According to Austin it sees a 62% open rate, a 16% reply rate, and a 6% “action rate.
What’s going on?
As we’ve seen above, consumers want to know what to expect when they open an email. Even if you can successfully increase open rates with a creative subject line, response rates will drop when your subscribers realize the content of the email isn’t relevant or interesting to them.
When it comes to email subject lines, there is no debate: it’s better to be clear than clever.
8. Emojis might help your subject line stand out, but they could be hurting open rates…
Emojis are commonly used to draw attention to emails in busy inboxes, but do they actually increase open rates? Creative consultancy agency YellowBall ran a series of tests to find out. Overall, subject lines with emojis were opened an average of 2% less often than those without them.
This is in stark contrast to the inspiration for YellowBall’s test – research by Experian – that showed a 50% increase in open rates for subject lines that included emojis.
What’s going on?
There are at least a couple of factors at work here. Demographics are going to play a part in how responsive an audience is to emojis. Older or corporate audiences are more likely to be turned off by the presence of emojis in subject lines than their younger counterparts, who are likely to be more accustomed to the role emojis play in digital communication.
They can also render incorrectly (appearing like this: □) and potentially flag spam filters (though as we touched on earlier, this is less of a concern than it used to be).
Some pros. Some cons. Unsure? Test them for yourself and your brand.
9. …Unless it’s a clock, used alongside a limited-time offer
Creative agency Brandwidth conducted an A/B test on the subject line of an email for their client, Royal Caribbean International.
Both subject lines included the text “Hurry – Offer ends in 3 days!” but variation B included an emoji of a clock alongside it.
Variation B saw a 3% increase in open rates and a 15% increase in click-throughs.
What’s going on?
Another factor playing into the effectiveness of emojis in subject lines is their contextual relevance. Don’t use them purely to draw your recipients’ attention to your email – use them as a means of reinforcing the message you want to get across.
10. Frequent emails won’t necessarily decrease open rates
This isn’t strictly about subject lines – although they did play a key part in the tests – something I’ll touch on below.
If you email your subscribers too often, they will become annoyed and unsubscribe from your list, and that will decrease your open rates.
Makes sense, right? Finding the right frequency is a fine line.
Image Source
However, tests carried out by Dave Huffman showed that this isn’t necessarily the case.
Huffman’s research involved sending one email a week for six weeks. He expected open rates to fall slightly each week, but they actually remained decidedly similar throughout the campaign.
What’s going on?
While there’s undoubtedly a limit to how often you can email your subscriber list before they’ll hit the “unsubscribe” button, if you’re consistently sending content worth seeing, and using direct, descriptive subject lines (another clear winner from Huffman’s research) this limit may be much higher than you think.
11. Putting yourself ahead of the customer can get you what you want – if you word it right
An A/B test for Money Dashboard set out to uncover whether making the recipient or their business the focus of the subject line would get better results.
Subject line A: “What’s your opinion here [first name]”
Subject line B: “Please put us out of our misery”
The winner: Subject line B (104.5% increase in opens for inactive subscribers and a 30.5% increase in active subscribers, as well as a 228.5% increase in clicks for inactive subscribers and a 103.3% increase in clicks for active subscribers)
What’s going on?
Subject line B actually goes against some of the takeaways listed above (not to mention best practices for writing subject lines in general). It’s very ambiguous, and the way it puts the sender ahead of the recipient is bordering on narcissistic.
This alone teaches us something – that there are few definitive lessons to take away from A/B testing emails (or anything for that matter). Every company, audience and email is different, and as such, what works for someone else might not work for you, and what works best once may not work best the next time.
However, the use of the word “misery” here is unusual. It helps attract attention and from there, invokes an emotional, empathetic response in the audience.
“There are limitless possible emotional combinations of language available. Knowing which work best for your audience is impossible without testing. The opportunity is to quantify emotion, then optimize based on the results, then profit from better subject lines.” ~Parry Malm
This shows that content which is all about you can work as well, if not better, than content which focuses on your audience – at least if you word it in a way that taps into your audience’s emotions.
12. Dropping the corporate-speak is effective, even when the subject matter is serious
Obama must have one of the most-talked-about email campaigns of all time. His subject lines scorned all best practices, yet got great results. Being Obama would have helped here. A lot. But relentless A/B testing played a big part too.
Specifically, his marketing team found that casual, personal, and friendly subject lines worked best – even when the content of the email itself was somewhat more serious.
Surprisingly, their best performing subject line overall was just “hey.”
What’s going on?
Simple – these subject lines replicate what you might receive from a friend or family member. Keep it simple, friendly, and real. Write like the real human being you are.
13. Using all caps isn’t always bad
The list of subject line don’ts almost always includes using all-caps. Another test from The Tumbleweed Tiny House Company pitted a lower-case subject line against the exact same wording in upper-case.
Subject line A: “SOLAR LIVING GOES TINY”
Subject line B: “Solar living goes tiny”
The winner: Subject line A (18% more opens)
What’s going on?
The winning subject line is bold and stands out. Just bear in mind that using all caps in a subject line can trigger spam filters. Then again, as we’ve already seen, this risk is mitigated if you’re consistently sending quality emails.
To be absolutely sure, it might be worth trying this one out on a small portion of your best subscribers (i.e. those who read your emails most regularly).
14. Separating the themes of your email could be more effective than trying to summarize them into a single sentence
Estate agency Spicerhaart tested two very similar versions of the same subject line. One described the contents of the email in a single sentence; the other split it into three distinct parts.
Subject line A: “Mill View | Join us for our pre-launch weekend to find out more about these exciting new homes”
Subject line B: “Mill View | Pre-launch weekend | Quality homes affordably priced”
The winner: Subject line B (74% more opens)
What’s going on?
This is another really simple one to answer – subject line B is easier to skim, and as such, is much less likely to be overlooked and ignored.
15. Plain, boring and descriptive might be the most effective subject line style of all
This last one isn’t from a single A/B test – it’s the result of an analysis of 40 million emails sent through MailChimp. I wanted to wrap up with it, however, for two reasons:
- The sample size of the data
- The fact that the results surprised me more than anything else on this list
So, out of 40 million emails, the best performing subject line was simply…
“[COMPANY NAME] Sales & Marketing Newsletter.”
What’s going on?
We have to consider the specific circumstances surrounding these emails – they may have been sent to a very small, but highly-targeted list of recipients who always open emails from the sender in question.
That said, we can’t ignore the fact that they do something that’s helped many winning subject lines on this list get the edge: they clearly describe the subject of the email.
This goes to prove once again that simply being upfront and honest is all your audience really wants (a lesson that can coincidentally be applied to all elements of running and marketing a business).
Just tell them what’s in the email. Could that be the golden rule? Test your audience and find out.
“Almost any questions can be answered, cheaply, quickly and finally, by a test campaign. And that’s the way to answer them – not by arguments around a table. Go to the court of last resort – the buyers of your product.” ~Claude Hopkins
Test different words and phrases. Negative vs positive. With or without emojis. Question vs statement. Personalization. Urgency, scarcity, ambiguity, and more. Test. Everything.
What’s the ideal subject line length for your audience? Test.
What are the most powerful words and phrases for your audience? Test.
Test your email campaigns. Read about the tests others are doing (but remember that their audience and your audience could respond differently).
As Ogilvy famously said, never stop testing, never stop improving. Your subject line gets you in. Make sure it’s the best it can be.
Image: Unsplash
Continue reading
Social
