A/B testing for email campaign content

  • 8
  • Idea
  • Updated 2 years ago
  • Implemented
  • (Edited)
Love the ability to split test subject lines, but would additionally love the ability to easily break a group into two -- each receiving a different newsletter for A/B testing.


As it stands, if I want to A/B test my newsletters I have to make a search, export it to my computer at the last possible second to get maximum subscribers, copy and paste half the group into a new excel sheet, THEN import these two new excel sheets back into Emma and THEN schedule the two competing newsletters for the exact same time, with the same split test subject lines. That's a lot of excel work to break a group in half!


Anything that would enable us to test our newsletters easier to find maximum efficiency would be fantastic!


Thanks
Photo of Mountaineers Books

Mountaineers Books

  • 11 Posts
  • 2 Reply Likes

Posted 6 years ago

  • 8
Photo of Ben Vance

Ben Vance, Alum

  • 328 Posts
  • 14 Reply Likes
Thanks for sharing your idea! Just out of curiosity, can you tell me a little more about why you're wanting to test multiple versions of a newsletter as opposed to just subject lines? We'd love to know your thoughts there!
Photo of Mountaineers Books

Mountaineers Books

  • 11 Posts
  • 2 Reply Likes
Of course Ben.

One test I'm manually trying right this week with an upcoming newsletter is long, text heavy newsletter vs image focused, short newsletter. I am finding lately that some of my short, single image calls-to-action newsletters are driving large engagement to purchase, but my longer text emails I've sent lately have had almost no sales. All having the same open rate, all used split tested subject lined. So, I've created my text heavy newsletter, and then took the same content from that and broke it into three call-to-action images. Half the list will receive a long text email, the other the shorter image and I'll see which gets more clicks, and most importantly, sales.

But on a smaller scale I'd love to put the same kind of A/B testing found in websites in newsletters. Links as the "order now" call-to-action in one newsletter versus image buttons. Adding direct add to cart buttons in one newsletter vs asking people to click to the page and add to cart on our site. Changing the offer wording. Small things, but things that if you test in each of your newsletters will better help you understand what your subscribers really react on, and what they ignore.

Would love it if it was offered. Would save me a LOT of time doing this manually and I think other Emma users would be thrilled at the option.

Thanks for listening to my idea.

Ashley
Photo of Katie Lewis

Katie Lewis, Official Rep

  • 3414 Posts
  • 309 Reply Likes
Hi Ashley,

It's been so interesting to read how folks are using our products in ways we didn't anticipate. (I found this list online that might interest you about other marketing tests, getting down to the nitty-gritty.) I checked in with our product owners, and we aren't currently pursuing what you've posited. But I wanted to address what sounds like your main concern: whether your image-focused or text-heavy newsletters are more successful.

Once you have a list of customers who want to receive your emails, the biggest hurdle is getting them to even open your emails. A survey conducted in the 2012 holiday season found that 60% of people said they deleted emails without reading them. That means your subject line must be engaging, and the RSVP email address and from name should be familiar.

Assuming you've already combed over your sending patterns -- not only the RSVP information but the days/times you're sending -- keep in mind that more of your recipients are now choosing mobile experiences for their shopping. I advise focusing your efforts on fine-tuning your campaigns' mobile experience instead of testing images vs. text. On my iPhone, I like a mix of images and text to get the message across, but I read more closely on my phone than my laptop, since there are fewer words per screen and less opportunity to skim. That being said, it's a pain to scroll through paragraph upon paragraph in a text-heavy campaign. And that being said, I'm only one user of many, and there's a bevy of habits tech-savvy researchers are likely examining.

Until those studies are out, your testing efforts are better focused on the arrangement of your content in a mobile environment and how it's presented (i.e. your calls to action) than making your campaigns about images or text. I'm going to update this topic's title to make it a little more searchable, but I'm glad to take a look at any specific campaigns, if you'd like to create a new topic here for some insight!
Photo of Mountaineers Books

Mountaineers Books

  • 11 Posts
  • 2 Reply Likes
Katie,

Thanks for your reply. What is the cost for updating our custom stationary (two versions) to be mobile friendly?

Thanks, Ashley
Photo of Katie Lewis

Katie Lewis, Official Rep

  • 3414 Posts
  • 309 Reply Likes
This is a great conversation that's separate from the main topic, so I created a new topic to continue the discussion. Please reference the new topic here: How much does it cost to make templates mobile-friendly?
Photo of Zehno Media

Zehno Media

  • 10 Posts
  • 1 Reply Like
I would just like to add my +1 for a full A/B testing feature - we've recently moved to Emma from Campaign Monitor, who does offer a way to do this - so that's actually what I was expecting when I saw that Emma offered split testing. Would be VERY useful. As is, we might have to go back to CM for those particular campaigns, doing it manually seems like it would take a ton of time.
Photo of Homemakers Furniture

Homemakers Furniture

  • 1 Post
  • 1 Reply Like
I would also like to add my +1 for this particular piece of functionality! While I certainly agree that testing subject, timing, etc. are important to our business to gauge success and make improvements-- I would like to make the case that A/B testing for content is KEY to testing other aspects of a mailing's success (and as mentioned in the previous posts, how a mailing contributes to conversions, the ultimate measure of success!). 64% of the list you linked to within the response would actually be achieved through A/B testing of email content (everything from tone, to text vs. images, font color, etc.).
Photo of Katie Lewis

Katie Lewis, Official Rep

  • 3414 Posts
  • 309 Reply Likes
Good morning! I checked in with a product owner to see if there have been any plan changes, but this idea still isn’t planned for now. We have some great features in the pipeline, though, so stay tuned!
Photo of Hands On Call

Hands On Call

  • 6 Posts
  • 1 Reply Like
We are looking for a way to accomplish this, too. Please keep us posted if this gets released. It's definitely a pain to export the list and then split it in half and re-import it. 
Photo of Katie Lewis

Katie Lewis, Official Rep

  • 3414 Posts
  • 309 Reply Likes
Hi there! I'd love to hear more about your goal, because you can definitely A/B test for subject lines, just not content, so you shouldn't need to manually split your list in half.
Photo of Clean Energy Collective

Clean Energy Collective

  • 25 Posts
  • 3 Reply Likes
In order to do a full A/B split, it has to be done manually now. Emma only allows 20% of the audience to be split.
Photo of Katie Damico

Katie Damico

  • 11 Posts
  • 1 Reply Like
We would definitely be interested in the ability to a/b split test different content (rather than just subject lines), as we have done this using other platforms and found it very useful. In the past we've done content testing to determine what imagery is most persuasive, or if length of message affects conversion rate - and we plan to continue to test these kinds of content issues, so an easier way to do so would be great!!

Any chance this functionality will be investigated in the near future?

Photo of Katie Lewis

Katie Lewis, Official Rep

  • 3414 Posts
  • 309 Reply Likes
Hi Katie! This work is still not planned for the next few months, and I’ll let you know if that changes. :)
Photo of Katie Lewis

Katie Lewis, Official Rep

  • 3414 Posts
  • 309 Reply Likes
Hi all! A Thursday update for you: I’m moving this from Not Planned to Under Consideration as we inch closer. There’s a good deal of work to be done here, and I’ll keep you updated with news as it comes.
Photo of McCoy's Building Supply

McCoy's Building Supply

  • 1 Post
  • 0 Reply Likes
Thanks Katie. This functionality would be very useful for us too. It's a pretty standard feature with enterprise level providers.
Photo of Jon

Jon

  • 1 Post
  • 1 Reply Like
Thanks Katie -

We recently signed up with an agency account based on the ability to split test (assuming content, not simply subject lines).

Unfortunately we got all the way to sending the email before we realized this functionality doesn't exist.

I'll add my vote for this functionality.
Photo of Sam

Sam

  • 1 Post
  • 0 Reply Likes
My +1, as well.  I have slightly different copy in my 2 emails, as well as different headlines.  Would love the ability to easily do a full content A/B test.
Photo of Infantino

Infantino

  • 1 Post
  • 1 Reply Like
There are nearly unlimited reasons to test content within the newsletter. Testing one type of discount vs. another, testing the impact of different marketing positioning statements, testing different call to actions... I love Emma, but not having this functionality is a big pain point in my email program.
Photo of Julie

Julie

  • 1 Post
  • 1 Reply Like
I thought I would post to see if there are any updates about content A/B testing? I was disappointed to see that only testing multiple subject lines is possible in Emma.
Photo of Katie Lewis

Katie Lewis, Official Rep

  • 3414 Posts
  • 309 Reply Likes
Hi Julie! The update is that we're still exploring what it would mean to our overall client base and to Emma if we build this, as well as what it might look like.

There are a number of factors that go into decisions to build, and we try to weigh all of those against each other so we can get a good sense of how a particular feature or product would fit into the Emma story. This is definitely still on our radar!
Photo of Restaurant Equippers

Restaurant Equippers

  • 27 Posts
  • 9 Reply Likes
Any more updates about content A/B testing? It would be very helpful!
Photo of Katie Lewis

Katie Lewis, Official Rep

  • 3414 Posts
  • 309 Reply Likes
Hello! We're still in the process of researching this, with the intention to deliver it later this year. We'll keep you posted with news as it develops. :)
Photo of Katie Lewis

Katie Lewis, Official Rep

  • 3414 Posts
  • 309 Reply Likes
Hello! This week, we released dynamic content, which means you can press Send on one mailing that delivers different versions to your contacts based on the data you're storing about them. We're already using it in our emails, and it's really exciting to know that our recipients are getting a message tailored to what we know about them without loads of extra work!

In this conversation, we've also talked about:
  • splitting and sending your message totally randomly instead of a specified message to targeted recipients
  • sending the best-performing version to the remaining addresses
We're still considering those two elements, though they aren't on the roadmap for this year right now. I'll let you know when that changes, and in the meantime, happy dynamic contenting!
Photo of Ackerman Cancer Center

Ackerman Cancer Center

  • 7 Posts
  • 0 Reply Likes
Hi Katie, 

Are there any plans yet to include split testing for anything other than subject lines? This would be extremely helpful. I understand that you have released dynamic content but we are sending out emails that I do not want to be targeted based on contact data - they need to be random. 

Thank you! 
(Edited)
Photo of Grey Stepp

Grey Stepp, Support Team Manager

  • 1700 Posts
  • 135 Reply Likes
Hey there folks!  

I just got the word from our product team that we are ready for some Beta testers for content testing!   

The new feature is only going to be available to customers on Pro, Plus, or Premium Packages and those with the New Emma Interface (with sidebar navigation) .  

If you're on one of these plans and would be interested in helping us test out this new feature, drop a comment on this thread and let me know.  We're really excited to get this product going!  

If you're still on the old interface or in an older plan and need to update, give us a shout and we'll be glad to take a look at getting you updated!   

Cheers!  

- Grey 
Photo of Missouri Botanical Gardens

Missouri Botanical Gardens

  • 2 Posts
  • 1 Reply Like
We would be interested in Beta testing A/B content. 
Photo of Grey Stepp

Grey Stepp, Support Team Manager

  • 1700 Posts
  • 135 Reply Likes
Thanks!  I've passed your information on to our product team and they will be reaching out soon with more information!
Photo of Missouri Botanical Gardens

Missouri Botanical Gardens

  • 2 Posts
  • 1 Reply Like
Thank you!
Photo of Grey Stepp

Grey Stepp, Support Team Manager

  • 1700 Posts
  • 135 Reply Likes
FYI, our Product team member who's handling these requests has been out, but we'll be getting in touch soon! 
Photo of Wings WorldQuest

Wings WorldQuest

  • 4 Posts
  • 2 Reply Likes
We would be interested in learning more about participating in the A/B testing beta.
Photo of Grey Stepp

Grey Stepp, Support Team Manager

  • 1700 Posts
  • 135 Reply Likes
Gotcha!   It looks like you guys are on one of our older plans and on the old interface.  Would you like for me to get you in touch with someone from our pricing team to go over plan options to get you on a plan that would have access to this feature?  
Photo of Wings WorldQuest

Wings WorldQuest

  • 4 Posts
  • 2 Reply Likes
No, that's OK. Thanks anyway. 
Photo of Mansfield Energy

Mansfield Energy

  • 2 Posts
  • 1 Reply Like
We would also be interested in trying the beta A/B testing.
Photo of Grey Stepp

Grey Stepp, Support Team Manager

  • 1700 Posts
  • 135 Reply Likes
Whoop!   I'll pass your information along to our product team and we'll be getting in touch soon!  
Photo of Grey Stepp

Grey Stepp, Support Team Manager

  • 1700 Posts
  • 135 Reply Likes
Official Response
Hi everyone!   

We've officially released A/B Content Testing for all Pro, Plus, and Premium accounts!  

Check out my Announcement here:
https://community.myemma.com/emma/topics/release-notes-a-b-content-testing

Also check out our blog post on it here:
http://content.myemma.com/blog/now-available-a-b-content-testing

And our RESOURCE CENTER documentation here:
https://support.e2ma.net/Resource_Center/Account_how-to/Content_split_testing

Cheers!  

- Grey