Rare intermittent write failures in Tempo
Incident Report for Grafana Cloud
Resolved
This incident has been resolved.
Posted Jan 23, 2025 - 23:04 UTC
Monitoring
Multiple issues were identified with our rollout process that caused instability on the write path.
We have pushed out three fixes that drastically reduce the number of dropped writes and make all failed writes retryable. We will continue to closely monitor this issue and make improvements where necessary.
Posted Jan 22, 2025 - 22:11 UTC
Update
We are continuing to diagnose and improve this situation. Fixes have been rolled out on Monday January 20th, and today, Wednesday January 22nd, to reduce drops during rollouts.
Posted Jan 22, 2025 - 19:52 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Jan 18, 2025 - 00:16 UTC
Update
We are continuing to investigate this issue.
Posted Jan 17, 2025 - 19:56 UTC
Investigating
Users may receive internet write failures on collectors stating:
"Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "traces", "name": "otlp/cloud", "error": "rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway)"

The agent/collector will automatically retry the write attempt and succeed, so there is no loss in data collection.
Posted Jan 17, 2025 - 19:56 UTC
This incident affected: Grafana Cloud: Tempo (AWS Australia - prod-au-southeast-1: Ingestion, AWS Brazil - prod-sa-east-1: Ingestion, AWS Canada - prod-ca-east-0: Ingestion, AWS Germany - prod-eu-west-2: Ingestion, AWS India - prod-ap-south-1: Ingestion, AWS Japan - prod-ap-northeast-0: Ingestion, AWS Singapore - prod-ap-southeast-1: Ingestion, AWS Sweden - prod-eu-north-0: Ingestion, AWS US East - prod-us-east-0: Ingestion, AWS US West - prod-us-west-0: Ingestion, Azure Germany - prod-eu-west-1: Ingestion, Azure Netherlands - prod-eu-west-3: Ingestion, Azure US Central - us-central2: Ingestion, GCP Australia - prod-au-southeast-0: Ingestion, GCP Belgium - prod-eu-west-0: Ingestion, GCP Brazil - prod-sa-east-0: Ingestion, GCP India - prod-ap-south-0: Ingestion, GCP Singapore - prod-ap-southeast-0: Ingestion, GCP UK - prod-gb-south-0: Ingestion, GCP US Central - prod-us-central-0: Ingestion).