RESOLVED: Database outage impacting a number of customers

  • 2
  • Problem
  • Updated 5 years ago
  • Solved
  • (Edited)
Archived and Closed

This conversation is no longer open for comments or replies and is no longer visible to community members. The community moderator provided the following reason for archiving: Resolved

Our infrastructure and development teams have noticed some problematic activity on one of our many databases. In this case a decision was made to take that database offline in order to safely and fully diagnose the issue. 

At this point, the outage impacts about 30% of our customers.  

Our team has begun working to restore the database and bring it back online as quickly as possible. Although significant progress has been made we do not yet have a reliable time to resolution. We will post an update here at, or before, 8:00pm. 
Photo of Ben Vance

Ben Vance, Alum

  • 328 Posts
  • 14 Reply Likes

Posted 5 years ago

  • 2
Photo of Ben Vance

Ben Vance, Alum

  • 328 Posts
  • 14 Reply Likes
Hi everyone.  Our developers are continuing to work on this, but at the present time we can’t say how quickly we’ll have things back to normal.  We’ll provide another update here in 3 hours (11pm CDT) or sooner if we have news.
(Edited)
Photo of Kyle Floyd

Kyle Floyd, Official Rep

  • 178 Posts
  • 26 Reply Likes
Our development team is continuing to work towards restoring the database that we pulled offline earlier today. We do not yet have a time frame for when we expect this work to be completed but we are well aware of how this unexpected downtime affects you, our customers, and are working as quickly as possible to bring Emma back online.

At this point, I do know that our developers will be working through the night or until full functionality of Emma is restored. That said, it may be several hours before our next update here. I will definitely post an estimated time to resolution as soon as I have that information available.
Photo of Kyle Floyd

Kyle Floyd, Official Rep

  • 178 Posts
  • 26 Reply Likes
I have great news.  Our server was brought back online roughly 30 minutes ago.

Here are some additional details about this database downtime that you may find important.

The unplanned outage only impacted one of our databases, so approximately 30% of our customers were affected. Relatively few customers were actively trying to edit or send a mailing during time of the outage, but there are a number of other activities Emma processes round-the-clock that were also affected. I’ve outlined the details around those events below.

Scheduled mailings: If you had a mailing scheduled between the hours of 2:00pm CDT on June 1 and 4:30am CDT on June 2, that mailing was delayed until roughly after 4:30am CDT on June 2, but it *did* send.

Previously sent mailings: During the outage, previously sent mailings were still visible to your audience in their inbox. But signups, clicks, webviews (view it online) and shares for already sent mailings would not have worked during this time. Those readers attempting any of those actions would have received an error.

Signup forms: Emma forms that are hosted on your site may not have worked during this time. Most likely the form simply didn't appear. If the form was able to load, the user attempting to complete the signup would have experienced an error.

Response Data: Mailing delivery, bounce data, open, click, and share tracking was not affected by this downtime. All response data is accurate and available in your account.

Again, this downtime impacted less than one-third of our customer base. We have taken further steps that we believe will prevent this from happening in the future, and we will explore additional options with our full team on Monday. In the meantime, we’ll continue to watch this and the other systems for any signs of a problem.

Please do post any questions below and we will be glad to add clarity.


(Edited)