We haven't seen any issues after the fix was deployed. We're actively monitoring the system and are in the process of taking steps to make sure this doesn't repeat.
While not ideal, it should make sure your queries succeed after a couple of retries and we've bumped our caches to make sure the queries to GCS are reduced.
We're now exploring how to fix this long-term.
Posted May 29, 2019 - 09:57 UTC
Identified
We've identified the issue and we're working on the fix. The issue is around GCS ratelimiting our requests and us not failing the requests fast enough.
Posted May 29, 2019 - 09:40 UTC
Investigating
We're currently seeing queries timeout and failing and are currently investigating the cause. Note that ingestion is unaffected and queries hitting only the recent data (last 2 days of data) should succeed.
Posted May 29, 2019 - 08:36 UTC
This incident affected: Metrics (GCP Belgium - prod-eu-west-0: Querying).