Partial Mimir Write Outage

Incident Report for Grafana Cloud

Resolved

This incident has been resolved.

Both read and write 5xx's and increased latency were experienced in the two periods:
23:56:15 to 00:32:45 UTC
00:55:30 to 01:36:15 UTC
Posted Jan 16, 2026 - 04:00 UTC

Monitoring

Customers should no longer experience issues.

We will continue to monitor and provide updates.
Posted Jan 16, 2026 - 02:27 UTC

Update

We are continuing to investigate this issue.
Posted Jan 16, 2026 - 01:35 UTC

Investigating

Users may experience intermittent 5xx errors when writing metrics, though retries may eventually succeed, which can lead to delayed or missing data.

We continue to investigate and will update when we have more to share.
Posted Jan 16, 2026 - 01:33 UTC

Monitoring

As of 00:28 UTC, we have observed improvement with the partial write outage. Customers should no longer experience issues with metrics ingestion.

We will continue to monitor and provide updates.
Posted Jan 16, 2026 - 00:59 UTC

Investigating

As of 23:57 UTC, our engineers became aware of an issue with prod-us-west-0 resulting in a partial write outage. Users may experience intermittent 5xx errors when writing metrics, though retries may eventually succeed, which can lead to delayed or missing data.

We continue to investigate and will update when we have more to share.
Posted Jan 16, 2026 - 00:28 UTC
This incident affected: Grafana Cloud: Prometheus (AWS US West - prod-us-west-0: Ingestion).