Engineering has released a fix and as of 10:30 UTC, customers should no longer experience hosted services issues. At this time, we are considering this issue resolved. No further updates.
Posted Jul 19, 2024 - 10:30 UTC
Update
As of 4:45 UTC, we have observed marked improvement with Prometheus, Loki, and Tempo components. We are continuing to monitor for signs of improvement in other components as this fix rolls out across our cloud service provider's services.
Posted Jul 19, 2024 - 04:49 UTC
Identified
Our cloud service provider has identified the cause of the outage and are applying mitigations. We expect to see signs of improvement as this fix rolls out across their services.
Posted Jul 19, 2024 - 02:22 UTC
Investigating
We are currently investigating issues with hosted services in us-central2.
Posted Jul 18, 2024 - 23:13 UTC
This incident affected: Profiles (Azure US Central - us-central2), Logs (Azure US Central - us-central2), OpenTelemetry Ingest Endpoint (Azure US Central - us-central2), AI and Machine Learning (Azure US Central - us-central2), Frontend Observability (Azure US Central - us-central2), Private Datasource Connect (PDC) (Azure US Central - us-central2), Grafana (Azure US Central - us-central2), Synthetic Monitoring (Azure US Central - prod-us-central2: API, Azure US Central - prod-us-central2: Public Probes), Metrics (Azure US Central - us-central2: Querying, Azure US Central - us-central2: Ingestion), Alerting and SLO (Azure US Central - us-central2: Rule Evaluation, Azure US Central - us-central2: Alertmanager and Rules Configuration API, Azure US Central - us-central2: Alertmanager), and Traces (Azure US Central - us-central2: Querying, Azure US Central - us-central2: Ingestion, Azure US Central - us-central2: Metrics Generator).