Prometheus writes, Logs, and Synthetic Monitoring in prod-eu-west-3 are degraded

Incident Report for Grafana Cloud

Resolved

This incident has been resolved.
Posted Mar 25, 2026 - 12:52 UTC

Update

This is also now impacting Logs and Synthetic Monitoring in prod-eu-west-3.

For Synthetic Monitoring, users might observe errors pushing check execution metrics, and this can eventually lead to missing data.
In addition, users might observe errors evaluating Synthetic Monitoring provisioned alert rule evaluations, and this can lead to missed alerts.

For Logs, there is no immediate impact on alerts, however, remote writes to Mimir is delayed which means users may see gaps in their recording rules.
Posted Mar 25, 2026 - 07:43 UTC

Investigating

We are moving this back to 'Investigating' as we are now observing a substantial drop in successful ingestion and increase in write path errors, and elevated rule evaluation latency and error. Reads are mostly fine. Our Engineering team is actively investigating this and we will provide further updates as our investigation progresses.
Posted Mar 25, 2026 - 07:04 UTC

Update

We have not observed any recent errors, but we will continue to monitor while we work with our CSP.
Posted Mar 24, 2026 - 21:23 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Mar 24, 2026 - 09:19 UTC

Investigating

We are currently experiencing degraded writes for mimir-prod-22 in prod-eu-west-3 since 08:45Z.
Posted Mar 24, 2026 - 09:08 UTC
This incident affected: Grafana Cloud: Loki (Azure Netherlands - prod-eu-west-3), Grafana Cloud: Synthetic Monitoring (Azure Netherlands - prod-eu-west-3: API, Azure Netherlands - prod-eu-west-3: Public Probes), and Grafana Cloud: Prometheus (Azure Netherlands - prod-eu-west-3: Ingestion).