Engineering has released a fix and as of 05:30 UTC, customers should no longer experience data delays/loss in the cortex-prod-06 cell in the prod-us-central-0 cluster. At this time, we are considering this issue resolved.
Posted Sep 25, 2023 - 05:31 UTC
Update
We're observing data-loss that is partial and limited to the tenants in the cortex-prod-06 cell in the prod-us-central-0 cluster. Our Engineering team is still investigating the issue and we will provide updates accordingly.
Posted Sep 25, 2023 - 05:03 UTC
Investigating
As of 1:00 UTC, we became aware of an issue with a delay in recent data for aggregated series being shown on dashboards and other query results in the prod-us-central-0 region. Engineering is actively engaged and assessing the issue. We will provide updates accordingly.
Posted Sep 25, 2023 - 03:50 UTC
This incident affected: Grafana (GCP US Central - prod-us-central-0) and Metrics (GCP US Central - prod-us-central-0: Querying, GCP US Central - prod-us-central-0: Ingestion).