Loki writes impacted for a small percentage of traffic

Incident Report for Grafana Cloud

Resolved

Details:
From 19:19UTC to 19:45UTC the following actions were impacted:
* A small percentage of writes (< 3%) were timing out
* All rule evaluations of alerting rules were timing out
* A small percentage of queries (< 4%) were timing out

From 19:45UTC to 20:56UTC:
* notifications were failing to deliver

This issue has now been resolved.

Posted May 26, 2021 - 20:25 UTC

Monitoring

Some cluster maintenance has impacted writes for a small percentage of traffic, rule evaluations on logs were also impacted.

A fix has been implemented, and results are being monitored.

Posted May 26, 2021 - 19:53 UTC

This incident affected: Logs (GCP US Central - prod-us-central-0).