Grafana Cloud intermittent loadbalancer errors

Incident Report for Grafana Cloud

Resolved

The implemented fix has resolved the issue. All systems are operational.
Posted Jan 12, 2022 - 18:35 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jan 12, 2022 - 14:57 UTC

Update

We are still continuing to investigate this issue.
Posted Jan 12, 2022 - 14:09 UTC

Update

We are continuing investigating the issue with our vendors.
Posted Jan 12, 2022 - 13:19 UTC

Update

We are continuing to investigate this issue.
Posted Jan 12, 2022 - 11:50 UTC

Update

The incident is impacting other services on US-Central.
Posted Jan 12, 2022 - 10:58 UTC

Investigating

We are observing intermittent loadbalancer errors on Hosted Logs, resulting in 502 errors in queries and when pushing logs to Hosted Logs.
We are currently investigating the issue.
Posted Jan 12, 2022 - 10:36 UTC
This incident affected: Website and Support (Grafana.com), Cost Management and Billing, Logs (GCP US Central - prod-us-central-0), Metrics (GCP US Central - prod-us-central-0: Querying, GCP US Central - prod-us-central-0: Ingestion), and Traces (GCP US Central - prod-us-central-0: Querying, GCP US Central - prod-us-central-0: Ingestion).