Flowdock Down
Incident Report for Flowdock
Postmortem

There was a major incident on Flowdock of about 32 hours (between April 21, 1:30 UTC to April 22, 21:30 UTC) leading to the unavailability of the application for about 22 hours (”Outage Window”), and in some cases, loss of data as well as third-party organizations’ individuals being able to potentially access other organizations’ collaboration flow during the “Incident Window” (between April 21, 13:30 UTC to 21:30 UTC, and April 22, 4:30 UTC to 6:30 UTC).

This document provides a complete description of the incident, details of the root cause analysis, and the approaches put in place to prevent such occurrences in the future.

Posted May 14, 2020 - 15:59 UTC

Resolved
This incident has been resolved.
Posted Apr 23, 2020 - 07:31 UTC
Monitoring
Flowdock Users,

We are happy to inform you that Flowdock collaboration capabilities have been fully restored and the application is now online. In our effort to restore and verify the services and associated data, we have established that the outage and occasional erroneous behavior with some collaboration flows were related to a technical error and not an external attack on our systems.

We regret to inform you that due to technical limitations involved with the restoration of data from the outage window, we could not preserve the messages, flows, and other associated Flowdock activities from your organization, between April 21, 13:30 UTC to 21:30 UTC, and April 22, 4:30 UTC to 6:30 UTC. The activities from this time window will need to be re-created by the respective users.

Now that Flowdock is fully operational, a complete root cause analysis of the situation is our top priority. We will work on ensuring that appropriate system and process improvements are put in place to avoid such outages in the future.

We thank you for your patience during this time and apologize again for any inconvenience that may have been caused by this outage to you and your teams.
Posted Apr 22, 2020 - 20:52 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Apr 22, 2020 - 20:26 UTC
Update
Our teams have restored the technical services and are working on the last checks for verification. Once all the final checks are completed, we will bring Flowdock back online.
Posted Apr 22, 2020 - 20:25 UTC
Update
Flowdock Users,

We apologize for the inconvenience that may have been caused to you and your teams due to the outage of Flowdock collaboration capabilities in the past 24 hours. We want to assure you that our technical teams are on top of the issue and we are working round the clock to bring the systems back up and mitigate any potential adverse effects in the meantime. At the time of sending this notification, our teams have restored the technical services and a full verification is underway.

We would also like to inform you that the current outage and occasional erroneous behavior in some collaboration flows are caused by a technical error and not an attack on our systems.

As we make further progress and bring the services back online, we will provide an update every two hours through the Flowdock status page - http://status.flowdock.com/
Posted Apr 22, 2020 - 18:40 UTC
Update
We are continuing to work to restore Flowdock functionality. We fully appreciate the difficulty this long-running outage has caused, and your patience as we work around-the-clock to resolve it.
Posted Apr 22, 2020 - 14:54 UTC
Investigating
We are continuing to work to restore Flowdock functionality.

Following major issues with login failures, missing flows and 1:1s, our Operations team has been diligently investigating this issue. During the investigation, it was determined that the Flowdock application continued to degrade and cause more issues, so the service was stopped while we work to resolve the problem and bring Flowdock back up.

We wholeheartedly understand the impact this has on our customers and empathize with the difficulty trying to communicate without it - we hate being without it too! We push on to get Flowdock back up as soon as possible, and very much appreciate your patience.

If any team members are asking for updates, please have them subscribe to https://status.flowdock.com for any updates.
Posted Apr 22, 2020 - 07:08 UTC
This incident affected: App, Network, and Storage.