Multiple issues were identified with our rollout process that caused instability on the write path. We have pushed out three fixes that drastically reduce the number of dropped writes and make all failed writes retryable. We will continue to closely monitor this issue and make improvements where necessary.
Posted 30 days ago. Jan 22, 2025 - 22:11 UTC
Update
We are continuing to diagnose and improve this situation. Fixes have been rolled out on Monday January 20th, and today, Wednesday January 22nd, to reduce drops during rollouts.
Posted 30 days ago. Jan 22, 2025 - 19:52 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted 1 month ago. Jan 18, 2025 - 00:16 UTC
Update
We are continuing to investigate this issue.
Posted 1 month ago. Jan 17, 2025 - 19:56 UTC
Investigating
Users may receive internet write failures on collectors stating: "Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "traces", "name": "otlp/cloud", "error": "rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway)"
The agent/collector will automatically retry the write attempt and succeed, so there is no loss in data collection.
Posted 1 month ago. Jan 17, 2025 - 19:56 UTC
This incident affected: Grafana Cloud: Tempo (AWS Australia - prod-au-southeast-1: Ingestion, AWS Brazil - prod-sa-east-1: Ingestion, AWS Canada - prod-ca-east-0: Ingestion, AWS Germany - prod-eu-west-2: Ingestion, AWS India - prod-ap-south-1: Ingestion, AWS Japan - prod-ap-northeast-0: Ingestion, AWS Singapore - prod-ap-southeast-1: Ingestion, AWS Sweden - prod-eu-north-0: Ingestion, AWS US East - prod-us-east-0: Ingestion, AWS US West - prod-us-west-0: Ingestion, Azure Germany - prod-eu-west-1: Ingestion, Azure Netherlands - prod-eu-west-3: Ingestion, Azure US Central - us-central2: Ingestion, GCP Australia - prod-au-southeast-0: Ingestion, GCP Belgium - prod-eu-west-0: Ingestion, GCP Brazil - prod-sa-east-0: Ingestion, GCP India - prod-ap-south-0: Ingestion, GCP Singapore - prod-ap-southeast-0: Ingestion, GCP UK - prod-gb-south-0: Ingestion, GCP US Central - prod-us-central-0: Ingestion).