Hosted Metrics - Increased ingestion latency

Incident Report for Grafana Cloud

Resolved

A single cluster node (Kafka) was experiencing issues. All writes are only confirmed when all replicas have committed the write to disk. As writes to this one node where taking longer, ingestion latency increased for a large portion of ingest traffic. Once performance of the node hit built in safety limits it was automatically removed from the cluster.

Posted Nov 06, 2018 - 04:20 UTC

Investigating

We are experiencing degraded metric ingestion performance. Users may notice timeouts when publishing metric data. Engineers are investigating the problem.

Posted Nov 06, 2018 - 03:52 UTC