Delays in message delivery
Incident Report for Flowdock
The issue where messages are occasionally delivered slowly for a few minutes (where you see the "Message not delivered, still trying") that has been happening quite often recently are due to a rather nasty bug in Mongodb (our message database). The bug causes even simple queries to sometimes use completely wrong and irrational indices, which then take extremely long time to complete and that slows down the message delivery service in a way that even the priority messages (normal chat messages) are affected.

We have a temporary workaround for this which has been in place for about a week now and has worked well, but unfortunately there was one additional case that it didn't cover and that still caused a short hiccup today. We'll work around that case as well.

The problem is supposedly fixed in a newer major Mongodb release which we are currently testing in our QA. The plan currently is to roll it to production by the end of next week. After that the problem should be permanently resolved.

Again, we are sorry for the inconvenience these issues have caused.
Posted Nov 19, 2015 - 16:52 UTC