The Kafka cluster went out of sync because of a bug in zookeeper, which caused the partition lag of many Metrictanks to increase, which after a while made them become non-ready. Once they were non-ready we could not handle many of the customer queries anymore.
HM-US-West2 customer queries were not served during a time span of about 1h.
2019.01.29 12:21 (UTC): Issue causes zk pods to lose connection between each other
2019.01.29 12:21 (UTC): As a result some of the kafka pods stop replicating
2019.01.29 12:53 (UTC): We restart the kafka-5 (active controller)
2019.01.29 13:05 (UTC): We restart the kafka-3 / kafka-12 pods (the two out of sync)
2019.01.29 13:14 (UTC): All the kafka pods fully replicated again and everything recovered
The connection between the zookeeper pods timed out for an unknown reason. This caused another zookeeper pod to become the new leader. Usually this shouldn’t be a problem, but in this case it caused us to hit a bug (https://issues.apache.org/jira/browse/ZOOKEEPER-2985). As a result some Kafka pods got out of sync, which caused some of the Metrictanks consuming from these pods to not ingest data on some partitions anymore, so their partition lag increased. Once the partition lag reached a certain threshold they declared themselves as non-ready.
We first restarted the kafka cluster’s active controller (kafka-5), and then the two kafka pods that failed to replicate their partitions (kafka-3, kafka-12).