We continue to observe a continued period of recovery. As of 00:08 UTC, we are considering this issue resolved. No further updates.
Posted Nov 18, 2025 - 00:10 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 17, 2025 - 23:27 UTC
Update
Our teams have been alerted that Synthetic Monitoring will also be affected by this outage.
Users may see gaps in their Synthetic Monitoring metrics as well as missed alerts as a result of this.
We continue to investigate and will provide further updates as they become available.
Posted Nov 17, 2025 - 22:55 UTC
Update
The investigation has revealed that metric ingestion is also affected, including log-generated recording rules. We are continuing to investigate the root cause and will provide further updates as more information becomes available.
Posted Nov 17, 2025 - 22:47 UTC
Update
The investigation has revealed that metric ingestion is also affected, including log-generated recording rules. We are continuing to investigate the root cause and will provide further updates as more information becomes available.
Posted Nov 17, 2025 - 22:47 UTC
Update
Span metrics have also been identified as affected. We are continuing to investigate the root cause and will share further updates as more information becomes available.
Posted Nov 17, 2025 - 22:33 UTC
Investigating
We are investigating elevated Mimir read errors beginning at approximately 21:57 UTC.
The errors are technically retriable, but most retries are unlikely to succeed at this time. This may result in failed or delayed query responses for some users.
Engineering is actively investigating the root cause and working to restore normal read performance.
We will provide further updates as we learn more.
Posted Nov 17, 2025 - 22:22 UTC
This incident affected: Grafana Cloud: OTLP Endpoint (AWS US East - prod-us-east-0), Grafana Cloud: Loki (AWS US East - prod-us-east-0), Grafana Cloud: Synthetic Monitoring (AWS US East - prod-us-east-0: Public Probes), Grafana Cloud: Graphite (AWS US East - prod-us-east-0: Querying, AWS US East - prod-us-east-0: Ingestion), Grafana Cloud: Prometheus (AWS US East - prod-us-east-0: Querying, AWS US East - prod-us-east-0: Ingestion), and Grafana Cloud: Tempo (AWS US East - prod-us-east-0: Metrics Generator).