RFO: 2018-09-20 us-west cluster overload hindering ingestion

Summary

On Thursday, september 20th we experienced an overload of our data pipeline in our us-west hosted metrics cluster. This impacted ingestion (increased latency) for a majority of customers on this cluster as their data could not be sent to us in a timely fashion, resulting in data coming in with delays up to several minutes, and for some customers, up to about half an hour and false alerts.

Details

Thursday the 20th was supposed to be yet another day of minor upgrades without impact.
(as announced on our statuspage)

However it went wrong in 2 phases, and for two reasons:

• 11:15 UTC: one of the instances being updated exerted more load on the platform than it should have,
  and this resulted in a relatively minor overload of the ingestion pipeline, and some relatively small data delays.
  Ingestion times went from their usual mean of +- 200ms and max ~2 seconds to mean of ~1 second and max
  several seconds, resulting in data delays in the order of 10 seconds to a minute.
  Our monitoring did not pick this up as the difference was too subtle. The cause here was a concurrency setting for the
  upgrade that was aggressive, combined with our kafka cluster which was too close to its limits.
  (an upgrade puts more load on our kafka cluster).
• 12:00 UTC: due to a bug in our cluster upgrade automation (specifically a race condition in the interaction with kubernetes)
  the number of server instances upgrading concurrently exceeded defined limits by a factor of 2.
  This overwhelmed the cluster leading to severely degraded ingestion performance. Average ingestion latency peaked to
  10 seconds and more, and many attempts to publish data timed out. As a result, customers would have experienced a delay
  in data being available for queries. The length of the delay for each customer would have varied based on their specific
  carbon-relay-ng settings and metrics volume. This data delay may have lead to customer monitoring systems generating false alerts.

By manually interfering with our cluster automation, we were able to slow down the upgrade, reduce impact and restore
service. So customer's carbon-relay-ng instances started transmitting the buffered
data and had their delay reduced to 0 by between 12:50 and 13:15 UTC.
Generally, no data was lost except for a customer who had a misconfigured carbon-relay-ng setup. (known issue that is on
that particular customer's roadmap to resolve).

Conclusion and TODO's

1. review and adjust deployment concurrency settings to prevent another overload. This was completed this week.
2. Fix bug in deployment tool. To date, we've found two problematic interactions with k8s and fixed them.
3. Provision more capacity for kafka cluster. This is also done now.
4. Improve internal processes around capacity monitoring and management.
   We are currently discussing a better process and will start dedicating more time to capacity management.

Report written by Dieter Plaetinck, Grafana Labs. 2018-09-27