RESOLVED: Database outage impacting a number of customers

  • 1
  • Problem
  • Updated 5 years ago
  • Solved
  • (Edited)
Archived and Closed

This conversation is no longer open for comments or replies and is no longer visible to community members. The community moderator provided the following reason for archiving: Resolved

At around 8:45pm CDT on Friday, May 16th our infrastructure and development teams began to notice some problematic activity on one of our many databases. In this case a decision was made to take that database offline in order to safely and fully diagnose the issue.

At this point, the outage impacts about 30% of our customers.

Our team has worked through the night to restore the database and bring it back online as quickly as possible. Although significant progress has been made we do not yet have a reliable time to resolution. We will post an update here at, or before, 9:00am.
Photo of Kyle Floyd

Kyle Floyd, Official Rep

  • 178 Posts
  • 26 Reply Likes

Posted 5 years ago

  • 1
Photo of Kyle Floyd

Kyle Floyd, Official Rep

  • 178 Posts
  • 26 Reply Likes
I am excited to share that I just received a "we're back" update from our developers.

I will post more in-depth details here as I have them.
Photo of Kyle Floyd

Kyle Floyd, Official Rep

  • 178 Posts
  • 26 Reply Likes
Here are some additional details about this database downtime that you may find important.

The unplanned outage only impacted one of our databases, so approximately 30% of our customers were affected. The problem occurred at a time of day (and time of week) when the volume of direct Emma usage is very low. That means relatively few customers were actively trying to edit or send a mailing at the time of the outage, but there are a number of other activities Emma processes round-the-clock that were also affected. I’ve outlined the details around those events below.

Scheduled mailings: If you had a mailing scheduled between the hours of 9:00pm CDT on May 16 and 8:00am CDT on May 17, that mailing was delayed until shortly after 8:00am CDT on May 17, but it *did* send.

Previously sent mailings: During the outage, previously sent mailings were still visible to your audience in their inbox. But signups, clicks, webviews (view it online) and shares for already sent mailings would not have worked during this time. Those readers attempting any of those actions would have received an error.

Signup forms: Emma forms that are hosted on your site may not have worked during this time. Most likely the form simply didn't appear. If the form was able to load, the user attempting to complete the signup would have experienced an error.

Again, this downtime occurred at a very low usage time for our system and impacted less than one-third of our customer base. We have taken further steps that we believe will prevent this from happening in the future, and we will explore additional options with our full team on Monday. In the meantime, we’ll continue to watch this and the other systems for any signs of a problem.

Please do post any questions below and we will be glad to add clarity.
Photo of Glaucoma Research Foundation

Glaucoma Research Foundation

  • 3 Posts
  • 0 Reply Likes
I can verify that the outage affected recipients of our email sent on Friday -- links in the email did not work on Friday night, but on Saturday morning the links were working again.
Photo of Kyle Floyd

Kyle Floyd, Official Rep

  • 178 Posts
  • 26 Reply Likes
Hi, I am glad that all of the links were working correctly again on Saturday morning. Were there any additional details I could provide for you here?