Massive difference between split test open rates and total open rate

  • 0
  • 1
  • Question
  • Updated 8 months ago
  • Answered
  • (Edited)
Archived and Closed

This conversation is no longer open for comments or replies and is no longer visible to community members. The community moderator provided the following reason for archiving: EOL CLEANUP

We tested 2 subject lines for our latest mailing. The variations have 84% and 75% open rates(!), but the open rate on the full mailing is just 30%. This means the open rate for the bulk of our audience not in the split test was only 17%. We ended the test after 30 min.

Did something go wrong here, or did we just get a wacky sample for the split test? Curious if this has happened to anyone else. Screenshots below.


Photo of Brian

Brian

  • 5 Posts
  • 0 Reply Likes

Posted 9 months ago

  • 0
  • 1
Photo of Bethany Woerner

Bethany Woerner, Champion

  • 40 Posts
  • 19 Reply Likes
Hey Brian,

That does seem strange. Perhaps you did just get a "wacky sample".

How long after you sent the mailing did you view your open rates? Also, I know you only waited 30 minutes, but what time of day did you send at?
Photo of Phillip M

Phillip M, Official Rep

  • 247 Posts
  • 20 Reply Likes
Hi Brian,

Those numbers aren't quite what I would expect to see either considering the overall performance of the mailing. I'll do some digging on my end and I'll be sure to jump back into the conversation when I have something conclusive to report. Thanks for your patience in the meantime.
Photo of Brian

Brian

  • 5 Posts
  • 0 Reply Likes
Thanks, Bethany and Phillip. The test started around 2:30 p.m., and I posted this a couple hours later. As of this moment, both test variations have 84.5% open rates while the full mailing has a 34% open rate.
Photo of Brian

Brian

  • 5 Posts
  • 0 Reply Likes
Phillip,

It happened again yesterday. For the mailing we sent about 24 hours ago, the split test open rates are currently 41% and 72% while the total open rate is 35%.

I look forward to hearing what you find!
Photo of Anthony Miranda

Anthony Miranda, Customer Support Specialist

  • 135 Posts
  • 16 Reply Likes
Hi there!

I wanted to jump in and give you an update while Phillip is out of the office. 

Thanks for giving us the extra example, there. We did some spot checking using the segment tool to check on the open metrics and the numbers are lining up with what we're seeing on the overview page, so it looks like those counts are correct. We are, however, seeing a discrepancy when the data is exported, so that is where we're going to continue digging into this.

As soon as there is an update, we'll be letting you know!
Photo of Brian

Brian

  • 5 Posts
  • 0 Reply Likes
Phillip and Anthony,

Checking in to let you know that this issue occurred for the 3rd time in a row on Friday. That mailing reports split test open rates of 73.81% and 60.71% but shows an overall open rate of 32%.

Any idea yet what is going on?

The open rates on the split tests (of all 3 mailings) have generally been much higher than our average open rates, so it may be simply that the data is not being reported correctly. Either way, we would certainly like to understand the issue and get a fix ASAP!

Thanks for your work on this issue.
Photo of Bethany Woerner

Bethany Woerner, Champion

  • 40 Posts
  • 19 Reply Likes
Brian—Are you still ending the test after 30 minutes?
Photo of Brian

Brian

  • 5 Posts
  • 0 Reply Likes
The test for the 2nd mailing went the full 4 hours. The 3rd one went about 2 hours.
Photo of Bethany Woerner

Bethany Woerner, Champion

  • 40 Posts
  • 19 Reply Likes
Okay, thank you. I was wondering if the first 30 minutes were just showing outliers, but if one of your tests went for the full four hours, then that must not be the case.
Photo of Anthony Miranda

Anthony Miranda, Customer Support Specialist

  • 135 Posts
  • 16 Reply Likes
Hi Brian,

First off, thanks for hanging in there while we spent some time researching this. We've tracked the multiple examples you've pointed out, but have been continuing to see that the numbers displayed in these response reports are correct. We've also confirmed that there have not been widespread reports of split test metrics being off, so things appear to be working as expected.

While I can completely understand the concern when you're seeing a discrepancy in open rates between the overall mailing and each of the variations, that just points to your subject lines being very effective in those tests! With a small sample size (relative to your overall group) it only takes a few opens to drastically alter the open rate of one variation. Your overall open rates for the mailings as a whole are all consistently above average, they just have a larger sample size and the percentages don't move as drastically.

I hope this helps shine some light on your mailing metrics! I'm confident that the information we're seeing here is correct, but let me know if you have any lingering questions.