The incident was caused by multiple ingesters being unavailable at the same time due to moving ingester pods between nodes. It's a regular operation, but in this particular case the ingester took an unexpected long time to restart which coincided with another ingester eventually restarting at the same time, causing an issue.
Posted Jul 03, 2025 - 14:31 UTC
Update
Cluster fully operational.
Posted Jul 03, 2025 - 10:10 UTC
Monitoring
We're currently monitoring health of the cluster since the outage was resolved and the issue was identified.
Posted Jul 03, 2025 - 10:09 UTC
Investigating
We faced an issue with cells in ap-south-1 Hosted Logs region. Between 8:55 and 9:07 UTC this region faced the complete read and write paths outage. Since then it fully recovered and services are fully operational again. We're investigating the root cause right now.
Posted Jul 03, 2025 - 09:23 UTC
This incident affected: Grafana Cloud: Loki (AWS India - prod-ap-south-1).