We consider the incident as resolved now. With regards to the cause - a slow physical partition of the backend database, which is used by the control plane of a critical component caused increased latency and occasional overloading with subsequent failing of the write path. Once writes switched to a different partition, the latency dropped and error rate went down.
Posted Oct 30, 2025 - 13:36 UTC
Monitoring
Elevated latency issue no longer happened since the last update, but we're still monitoring the situation. The write path is operational now for eu-west-3 region.
Posted Oct 30, 2025 - 12:42 UTC
Update
We're still actively trying to mitigate the root cause. Symptoms are recurring from time to time.
Posted Oct 30, 2025 - 11:58 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Oct 30, 2025 - 09:51 UTC
Investigating
Since 9:15 UTC, Hosted Logs tenants in eu-west-3 region are experiencing an elevated latency in the write path. Our team is looking to identify and resolve the issue.
Posted Oct 30, 2025 - 09:30 UTC
This incident affected: Grafana Cloud: Loki (Azure Netherlands - prod-eu-west-3).