Degraded query performance as a result of elevated read latency on the trace by id path

Incident Report for Grafana Cloud

Resolved

This incident has been resolved.
Posted Feb 20, 2025 - 16:02 UTC

Identified

The issue has been identified, and we are rolling out a fix.
Posted Feb 20, 2025 - 15:05 UTC

Update

We are continuing to investigate this issue.
Posted Feb 20, 2025 - 14:11 UTC

Update

We are continuing to investigate this issue.
Posted Feb 20, 2025 - 14:08 UTC

Investigating

We are currently investigating this issue.
Posted Feb 20, 2025 - 14:07 UTC
This incident affected: Grafana Cloud: Tempo (AWS Australia - prod-au-southeast-1: Querying, AWS Australia - prod-au-southeast-1: Ingestion, AWS Australia - prod-au-southeast-1: Metrics Generator, AWS Brazil - prod-sa-east-1: Querying, AWS Brazil - prod-sa-east-1: Ingestion, AWS Brazil - prod-sa-east-1: Metrics Generator, AWS Canada - prod-ca-east-0: Querying, AWS Canada - prod-ca-east-0: Ingestion, AWS Canada - prod-ca-east-0: Metrics Generator, AWS Germany - prod-eu-west-2: Querying, AWS Germany - prod-eu-west-2: Ingestion, AWS Germany - prod-eu-west-2: Metrics Generator, AWS India - prod-ap-south-1: Querying, AWS India - prod-ap-south-1: Ingestion, AWS India - prod-ap-south-1: Metrics Generator, AWS Japan - prod-ap-northeast-0: Querying, AWS Japan - prod-ap-northeast-0: Ingestion, AWS Japan - prod-ap-northeast-0: Metrics Generator, AWS Singapore - prod-ap-southeast-1: Querying, AWS Singapore - prod-ap-southeast-1: Ingestion, AWS Singapore - prod-ap-southeast-1: Metrics Generator, AWS Sweden - prod-eu-north-0: Querying, AWS Sweden - prod-eu-north-0: Ingestion, AWS Sweden - prod-eu-north-0: Metrics Generator, AWS US East - prod-us-east-0: Querying, AWS US East - prod-us-east-0: Ingestion, AWS US East - prod-us-east-0: Metrics Generator, AWS US West - prod-us-west-0: Querying, AWS US West - prod-us-west-0: Ingestion, AWS US West - prod-us-west-0: Metrics Generator, Azure Germany - prod-eu-west-1: Querying, Azure Germany - prod-eu-west-1: Ingestion, Azure Germany - prod-eu-west-1: Metrics Generator, Azure Netherlands - prod-eu-west-3: Querying, Azure Netherlands - prod-eu-west-3: Ingestion, Azure Netherlands - prod-eu-west-3: Metrics Generator, Azure US Central - us-central2: Querying, Azure US Central - us-central2: Ingestion, Azure US Central - us-central2: Metrics Generator, GCP Australia - prod-au-southeast-0: Querying, GCP Australia - prod-au-southeast-0: Ingestion, GCP Australia - prod-au-southeast-0: Metrics Generator, GCP Belgium - prod-eu-west-0: Querying, GCP Belgium - prod-eu-west-0: Ingestion, GCP Belgium - prod-eu-west-0: Metrics Generator, GCP Brazil - prod-sa-east-0: Querying, GCP Brazil - prod-sa-east-0: Ingestion, GCP Brazil - prod-sa-east-0: Metrics Generator, GCP India - prod-ap-south-0: Querying, GCP India - prod-ap-south-0: Ingestion, GCP India - prod-ap-south-0: Metrics Generator, GCP Singapore - prod-ap-southeast-0: Querying, GCP Singapore - prod-ap-southeast-0: Ingestion, GCP Singapore - prod-ap-southeast-0: Metrics Generator, GCP UK - prod-gb-south-0: Querying, GCP UK - prod-gb-south-0: Ingestion, GCP UK - prod-gb-south-0: Metrics Generator, GCP US Central - prod-us-central-0: Querying, GCP US Central - prod-us-central-0: Ingestion, GCP US Central - prod-us-central-0: Metrics Generator).