Major GCP Incident Affecting Multiple Grafana Cloud Components (Including AWS and Azure deployed Instances)

Incident Report for Grafana Cloud

Resolved

We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.
Posted Jun 12, 2025 - 21:51 UTC

Monitoring

Google's engineering team have identified the root cause and applied appropriate mitigations, customers should no longer experience performance issues with Grafana Cloud products. We will continue to monitor for recurrence and provide updates accordingly.
Posted Jun 12, 2025 - 20:55 UTC

Update

The scope of this incident may be affecting all components in Grafana Cloud regardless of deployment region. This includes Alerting amongst all other products.
Posted Jun 12, 2025 - 20:17 UTC

Update

Our CDN provider, Fastly, is also impacted by the GCP outage, and this prevents the loading of all of the static files required. Users already logged in and with locally cached assets are unaffected.
Posted Jun 12, 2025 - 19:54 UTC

Update

After initial investigation, the scope of this issue may lead to users experiencing degraded performance across all of Grafana Cloud products.
Posted Jun 12, 2025 - 18:56 UTC

Investigating

As of 17:55 UTC, we were alerted a significant outage involving one of our cloud providers. Users experiencing this issue may encounter inability to query or ingest data for metrics, logs, traces, and profiles. Users experiencing this issue may also not be able to create new Hosted Grafana instances.

Engineering is actively engaged and assessing the issue. We will provide updates accordingly.
Posted Jun 12, 2025 - 18:36 UTC
This incident affected: Profiles (GCP Australia - prod-au-southeast-0, GCP Belgium - prod-eu-west-0, GCP Brazil - prod-sa-east-0, GCP India - prod-ap-south-0, GCP Singapore - prod-ap-southeast-0, GCP UK - prod-gb-south-0, GCP US Central - prod-us-central-0), Integrations, Cost Management and Billing, Fleet Management, Incident Management and Response (IRM), AI and Machine Learning (GCP Australia - prod-au-southeast-0, GCP Belgium - prod-eu-west-0, GCP Brazil - prod-sa-east-0, GCP India - prod-ap-south-0, GCP Singapore - prod-ap-southeast-0, GCP UK - prod-gb-south-0, GCP US Central - prod-us-central-0), OpenTelemetry Ingest Endpoint (GCP Australia - prod-au-southeast-0, GCP Belgium - prod-eu-west-0, GCP Brazil - prod-sa-east-0, GCP India - prod-ap-south-0, GCP Singapore - prod-ap-southeast-0, GCP UK - prod-gb-south-0, GCP US Central - prod-us-central-0), Frontend Observability (GCP Australia - prod-au-southeast-0, GCP Belgium - prod-eu-west-0, GCP Brazil - prod-sa-east-0, GCP India - prod-ap-south-0, GCP Singapore - prod-ap-southeast-0, GCP UK - prod-gb-south-0, GCP US Central - prod-us-central-0), Logs (GCP Australia - prod-au-southeast-0, GCP Belgium - prod-eu-west-0, GCP Brazil - prod-sa-east-0, GCP India - prod-ap-south-0, GCP Singapore - prod-ap-southeast-0, GCP UK - prod-gb-south-0, GCP US Central - prod-us-central-0, GCP US Central - prod-us-central-5), Private Datasource Connect (PDC) (GCP Australia - prod-au-southeast-0, GCP Belgium - prod-eu-west-0, GCP Brazil - prod-sa-east-0, GCP India - prod-ap-south-0, GCP Singapore - prod-ap-southeast-0, GCP UK - prod-gb-south-0, GCP US Central - prod-us-central-0), Grafana (GCP Australia - prod-au-southeast-0, GCP Belgium - prod-eu-west-0, GCP Brazil - prod-sa-east-0, GCP India - prod-ap-south-0, GCP Singapore - prod-ap-southeast-0, GCP UK - prod-gb-south-0, GCP US Central - prod-us-central-0, GCP US Central - prod-us-central-3, GCP US Central - prod-us-central-4, GCP US East - prod-us-east-1), Synthetic Monitoring (GCP Australia - prod-au-southeast-0: API, GCP Australia - prod-au-southeast-0: Public Probes, GCP Belgium - prod-eu-west-0: API, GCP Belgium - prod-eu-west-0: Public Probes, GCP Brazil - prod-sa-east-0: API, GCP Brazil - prod-sa-east-0: Public Probes, GCP India - prod-ap-south-0: API, GCP India - prod-ap-south-0: Public Probes, GCP Singapore - prod-ap-southeast-0: API, GCP Singapore - prod-ap-southeast-0: Public Probes, GCP UK - prod-gb-south-0: API, GCP UK - prod-gb-south-0: Public Probes, GCP US Central - prod-us-central-0: API, GCP US Central - prod-us-central-0: Public Probes), Metrics (GCP Australia - prod-au-southeast-0: Querying, GCP Australia - prod-au-southeast-0: Ingestion, GCP Belgium - prod-eu-west-0: Querying, GCP Belgium - prod-eu-west-0: Ingestion, GCP Brazil - prod-sa-east-0: Querying, GCP Brazil - prod-sa-east-0: Ingestion, GCP India - prod-ap-south-0: Querying, GCP India - prod-ap-south-0: Ingestion, GCP Singapore - prod-ap-southeast-0: Querying, GCP Singapore - prod-ap-southeast-0: Ingestion, GCP UK - prod-gb-south-0: Querying, GCP UK - prod-gb-south-0: Ingestion, GCP US Central - prod-us-central-0: Querying, GCP US Central - prod-us-central-0: Ingestion), and Traces (GCP Australia - prod-au-southeast-0: Querying, GCP Australia - prod-au-southeast-0: Ingestion, GCP Australia - prod-au-southeast-0: Metrics Generator, GCP Belgium - prod-eu-west-0: Querying, GCP Belgium - prod-eu-west-0: Ingestion, GCP Belgium - prod-eu-west-0: Metrics Generator, GCP Brazil - prod-sa-east-0: Querying, GCP Brazil - prod-sa-east-0: Ingestion, GCP Brazil - prod-sa-east-0: Metrics Generator, GCP India - prod-ap-south-0: Querying, GCP India - prod-ap-south-0: Ingestion, GCP India - prod-ap-south-0: Metrics Generator, GCP Singapore - prod-ap-southeast-0: Querying, GCP Singapore - prod-ap-southeast-0: Ingestion, GCP Singapore - prod-ap-southeast-0: Metrics Generator, GCP UK - prod-gb-south-0: Querying, GCP UK - prod-gb-south-0: Ingestion, GCP UK - prod-gb-south-0: Metrics Generator, GCP US Central - prod-us-central-0: Querying, GCP US Central - prod-us-central-0: Ingestion, GCP US Central - prod-us-central-0: Metrics Generator).