Between 12:38 and 13:05 UTC, the GCP Belgium (prod-eu-west-0) region experienced a brief period of failed Hosted Loki reads and writes. During this time period, some users would have experienced 4xx/5xx errors when trying to either query for Cloud logs or when trying to write logs to Hosted Loki. Additionally, due to this issue, some users may also have experienced a Hosted Loki log loss that we are unable to backfill.
Posted Aug 22, 2023 - 14:23 UTC
Update
We are continuing to monitor for any further issues.
Posted Aug 22, 2023 - 14:02 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Aug 22, 2023 - 13:17 UTC
Investigating
We are currently investigating this issue. This currently affects instances in loki-prod-eu-west-0 region only.
Posted Aug 22, 2023 - 13:12 UTC
This incident affected: Grafana Cloud: Loki (GCP Belgium - prod-eu-west-0).