Engineering has started rolling out a fix as of 23:12 UTC. We will continue to monitor for recurrence and provide updates accordingly.
Posted Oct 01, 2025 - 23:14 UTC
Update
The fix is still in progress, but we have made good headway. Once things are finalized, we will begin rolling it out to impacted environments.
Posted Oct 01, 2025 - 20:09 UTC
Update
We are continuing to work on a fix for this issue.
Posted Oct 01, 2025 - 17:31 UTC
Identified
We have identified the issue and are working on a fix.
Posted Oct 01, 2025 - 16:20 UTC
Investigating
Since the 1st of October at 4am UTC time, we have received reports of Mute Timings and Active Intervals on Alerting module having degraded performance. We are actively working on fixing this issue. Impact includes receiving notification from alerts that should be muted.
Posted Oct 01, 2025 - 14:56 UTC
This incident affected: Grafana Cloud: Prometheus Rules and Alerts (AWS Australia - prod-ap-southeast-2: Alertmanager, AWS Brazil - prod-sa-east-1: Alertmanager, AWS Canada - prod-ca-east-0: Alertmanager, AWS Germany - prod-eu-west-2: Alertmanager, AWS UAE - prod-me-central-1: Alertmanager, AWS Singapore - prod-ap-southeast-1: Alertmanager, AWS Sweden - prod-eu-north-0: Alertmanager, AWS US East - prod-us-east-0: Alertmanager, AWS US West - prod-us-west-0: Alertmanager, Azure Netherlands - prod-eu-west-3: Alertmanager, Azure US Central - us-central2: Alertmanager, GCP Australia - prod-au-southeast-0: Alertmanager, GCP Belgium - prod-eu-west-0: Alertmanager, GCP Brazil - prod-sa-east-0: Alertmanager, GCP India - prod-ap-south-0: Alertmanager, GCP Singapore - prod-ap-southeast-0: Alertmanager, GCP UK - prod-gb-south-0: Alertmanager, GCP US Central - prod-us-central-0: Alertmanager).