Resolved
This incident has been resolved.
Posted 9 days ago. Apr 08, 2025 - 07:20 UTC
Update
We were alerted to this incident affecting Mimir in the prod-us-east-0 cluster as well.
Posted 9 days ago. Apr 08, 2025 - 06:06 UTC
Update
We are continuing to investigate this issue.
Posted 9 days ago. Apr 08, 2025 - 05:46 UTC
Investigating
As of 5:30am UTC, we were alerted to an issue with the following components and what users experiencing these issues may encounter:
- Authentication failing
- Alerting impacted, unable to evaluate rules
- Loki impacted, errors on read and write path
- SLO plugin is down in the prod-us-east-0 cluster, unable to access page
Engineering is actively engaged and assessing the issue. We will provide updates accordingly.
Posted 9 days ago. Apr 08, 2025 - 05:42 UTC
This incident affected: Grafana Cloud: Prometheus (AWS Australia - prod-ap-southeast-2: Querying, AWS Australia - prod-ap-southeast-2: Ingestion, AWS Brazil - prod-sa-east-1: Querying, AWS Brazil - prod-sa-east-1: Ingestion, AWS Canada - prod-ca-east-0: Querying, AWS Canada - prod-ca-east-0: Ingestion, AWS Germany - prod-eu-west-2: Querying, AWS Germany - prod-eu-west-2: Ingestion, AWS India - prod-ap-south-1: Querying, AWS India - prod-ap-south-1: Ingestion, AWS Japan - prod-ap-northeast-0: Querying, AWS Japan - prod-ap-northeast-0: Ingestion, AWS Singapore - prod-ap-southeast-1: Querying, AWS Singapore - prod-ap-southeast-1: Ingestion, AWS Sweden - prod-eu-north-0: Querying, AWS Sweden - prod-eu-north-0: Ingestion, AWS US East - prod-us-east-0: Querying, AWS US East - prod-us-east-0: Ingestion, AWS US West - prod-us-west-0: Querying, AWS US West - prod-us-west-0: Ingestion, Azure Netherlands - prod-eu-west-3: Querying, Azure Netherlands - prod-eu-west-3: Ingestion, Azure US Central - us-central2: Querying, Azure US Central - us-central2: Ingestion, GCP Australia - prod-au-southeast-0: Querying, GCP Australia - prod-au-southeast-0: Ingestion, GCP Belgium - prod-eu-west-0: Querying, GCP Belgium - prod-eu-west-0: Ingestion, GCP Brazil - prod-sa-east-0: Querying, GCP Brazil - prod-sa-east-0: Ingestion, GCP India - prod-ap-south-0: Querying, GCP India - prod-ap-south-0: Ingestion, GCP Singapore - prod-ap-southeast-0: Querying, GCP Singapore - prod-ap-southeast-0: Ingestion, GCP UK - prod-gb-south-0: Querying, GCP UK - prod-gb-south-0: Ingestion, GCP US Central - prod-us-central-0: Querying, GCP US Central - prod-us-central-0: Ingestion), Grafana Cloud: Loki (AWS Australia - prod-ap-southeast-2, AWS Brazil - prod-sa-east-1, AWS Canada - prod-ca-east-0, AWS Germany - prod-eu-west-2, AWS India - prod-ap-south-1, AWS Japan - prod-ap-northeast-0, AWS Singapore - prod-ap-southeast-1, AWS Sweden - prod-eu-north-0, AWS US East - prod-us-east-0, AWS US West - prod-us-west-0, Azure Netherlands - prod-eu-west-3, Azure US Central - us-central2, GCP Australia - prod-au-southeast-0, GCP Belgium - prod-eu-west-0, GCP Brazil - prod-sa-east-0, GCP India - prod-ap-south-0, GCP Singapore - prod-ap-southeast-0, GCP UK - prod-gb-south-0, GCP US Central - prod-us-central-0, GCP US Central - prod-us-central-5), and Grafana OnCall, Grafana Cloud: SLO.