We continue to observe consistent, nominal behavior with metric ingestion and querying in all US GCP regions. We are therefore considering this incident resolved, at this time.
No further updates.
Posted Oct 05, 2023 - 19:11 UTC
Monitoring
As of 18:05 UTC, we have applied mitigations and have observed full recovery of all metric ingestion and querying components. Users in the US GCP regions should no longer experience adverse conditions when interacting with Hosted Metrics.
We will continue to monitor for a period of 30 minutes to ensure continued performance.
Posted Oct 05, 2023 - 18:22 UTC
Update
As of 17:45 UTC, Engineering continues to work towards remediation and resolution of metric ingestion and querying in the US GCP regions, only. We will continue to provide updates.
Posted Oct 05, 2023 - 17:45 UTC
Update
As of 16:44 UTC, we have observed recovery in Hosted Grafana, only. At this time, users previously experiencing either an inability to access an instance should no longer experience issues.
We continue to actively work toward resolution for difficulties with querying and/or ingesting Metrics in the US GCP Regions, only. We will continue to provide updates.
Posted Oct 05, 2023 - 17:03 UTC
Investigating
At approximately 15:00 UTC our team noticed degraded reads, writes, and availability for all products in the us-central region. Our team is actively investigating this incident and will provide updates when they become available.
Posted Oct 05, 2023 - 15:38 UTC
This incident affected: OpenTelemetry Ingest Endpoint (GCP US Central - prod-us-central-0), Private Datasource Connect (PDC) (GCP US Central - prod-us-central-0), Profiles (GCP US Central - prod-us-central-0), Logs (GCP US Central - prod-us-central-0), AI and Machine Learning (GCP US Central - prod-us-central-0), Frontend Observability (GCP US Central - prod-us-central-0), Grafana (GCP US Central - prod-us-central-0, GCP US Central - prod-us-central-3, GCP US Central - prod-us-central-4), Metrics (GCP US Central - prod-us-central-0: Querying, GCP US Central - prod-us-central-0: Ingestion), Synthetic Monitoring (GCP US Central - prod-us-central-0: API, GCP US Central - prod-us-central-0: Public Probes), Alerting and SLO (GCP US Central - prod-us-central-0: Alertmanager and Rules Configuration API, GCP US Central - prod-us-central-0: Alertmanager), and Traces (GCP US Central - prod-us-central-0: Querying, GCP US Central - prod-us-central-0: Ingestion).