This issue has been resolved and we are currently monitoring.
Posted May 22, 2023 - 21:42 UTC
Monitoring
During a weekly update cycle, we implemented a modification to the Tempo system that inadvertently led to a significant increase in the number of series being produced by the Metrics Generator. This issue affected multiple regions for a duration of 6-11 hours. We have since identified the root cause and have deployed a fix to address it.
Posted May 22, 2023 - 10:00 UTC
This incident affected: Grafana Cloud: Prometheus (AWS US East - prod-us-east-0: Querying, AWS US East - prod-us-east-0: Ingestion, Azure US Central - us-central2: Querying, Azure US Central - us-central2: Ingestion, GCP Australia - prod-au-southeast-0: Querying, GCP Australia - prod-au-southeast-0: Ingestion, GCP Belgium - prod-eu-west-0: Querying, GCP Belgium - prod-eu-west-0: Ingestion, GCP UK - prod-gb-south-0: Querying, GCP UK - prod-gb-south-0: Ingestion, GCP US Central - prod-us-central-0: Querying, GCP US Central - prod-us-central-0: Ingestion).