Engineering has released a fix and as of 10:00 am UTC, customers should no longer experience any alerting issues.
Posted Jan 24, 2024 - 02:40 UTC
Identified
We have partially worked out the issue, however, we had to scale down deployment in prod-us-central-0 and prod-eu-west-0. As a result of that, the previously mentioned issues with alert annotations not showing up and other degraded alert features have been mitigated, however, users may have trouble working with the SLO or Cloud Logs Exporter apps (creating/updating/deleting/listing objects).
Posted Jan 23, 2024 - 02:53 UTC
Investigating
As of 1:00 UTC, we became aware of an issue with alerting in the prod-eu-west-0 and prod-us-central-0 clusters. Users experiencing this issue may encounter issues with alert annotations not showing up and other degraded alert features.
Engineering is actively engaged and assessing the issue. We will provide updates accordingly.
Posted Jan 23, 2024 - 01:29 UTC
This incident affected: Grafana Cloud: Prometheus Rules and Alerts (GCP Belgium - prod-eu-west-0: Rule Evaluation, GCP Belgium - prod-eu-west-0: Alertmanager and Rules Configuration API, GCP Belgium - prod-eu-west-0: Alertmanager, GCP US Central - prod-us-central-0: Rule Evaluation, GCP US Central - prod-us-central-0: Alertmanager and Rules Configuration API, GCP US Central - prod-us-central-0: Alertmanager).