RFO: 2018-03-09 Kafka instances becoming unavailable, blocking ingest

Summary

Multiple Kafka instances restarted because they ran out of space after a version update made them consume more space for the same data volume. Because so many brokers failed at the same time, a small number of messages were lost. When the brokers came back online the Metrictank instances tried to consume from offsets that did not exist anymore. When a requested offset does not exist in Kafka, the Metrictank instances have no option but to restart as they cannot reliably determine what offset they should consume from.

Resolutions

Add more data volumes to the Kafka instances to give them more space.

Timeline (UTC)

• 11:00 - 12:00 Dieter & Mauro update all kafka pods in HM-US-East to a newer version. The new version seems to be running fine and functioning as expected
• 16:16 Mauro receives a disk free alert, decides to wait and see because the same alert fired just a few hours earlier and then self-resolved
• 17:25 Multiple Kafka pods restart
• 17:35 Dieter & Mauro start looking at the issue, quickly identifying that the reason for the restarts is that the Kafkas are running out of space
• 17:47 - 18:20 Dieter & Mauro extend the storage volumes used by the Kafka pods, this has to be done one-by-one so it takes a while
• 18:25 All Kafka pods are back up and have resynced again
• 18:25 - 20:16 All the various MTs that have restarted need to catch up with their backlogs, for some customers this takes longer than for others

TODO's

• Fix the alerting so there are less false positives. Separate critical from less-critical alerts.