Increased Metric Write Latencies

Incident Report for Grafana Cloud

Resolved

This incident has been resolved.

Posted Jul 10, 2023 - 10:56 UTC

Update

We are no longer experiencing the issue with increased metric write latencies in our systems. Engineering continues to investigate and identify long term mitigation or resolution strategies.

We will continue to provide updates.

Posted Jul 07, 2023 - 12:22 UTC

Investigating

As of 15:15 UTC, we were alerted to increased write latencies in our Mimir prod-6 system, only. Engineering is aware and actively investigating. Users making use of remote_write may experience degraded performance (50x errors).

We will continue to provide updates as more information is shared.

Posted Jul 06, 2023 - 15:44 UTC

This incident affected: Metrics (GCP US Central - prod-us-central-0: Querying, GCP US Central - prod-us-central-0: Ingestion).