Services are restored. Root cause is still being investigated.
Posted Mar 19, 2024 - 22:58 UTC
Update
As of 19:45 UTC, we have noticed marked improvement in the majority of Cloud users making use of the Prometheus API for querying needs. A small subset a users may continue to have an adverse experience and may still encounter a 500 ReadObject error when trying to query with the Prometheus API.
Engineering continues efforts towards complete remediation. We will provide updates accordingly.
Posted Mar 19, 2024 - 20:14 UTC
Update
Engineering has begun to employ remediation methods in order to alleviate the issues with querying using the Prometheus API. Users should begin seeing full functionality restored soon.
We will provide further updates upon the completion of remediation efforts.
Posted Mar 19, 2024 - 17:22 UTC
Investigating
As of 13:07 UTC, we were alerted to many Grafana Cloud users experiencing issues leveraging the Prometheus API to query their metrics. Users encountering this issue will be presented with a 500 ReadObject error.
Engineering is actively involved and working to remediate. We will monitor and provide updates.
Posted Mar 19, 2024 - 15:40 UTC
This incident affected: Grafana Cloud: Prometheus (AWS Australia - prod-au-southeast-1: Querying, AWS Brazil - prod-sa-east-1: Querying, AWS Canada - prod-ca-east-0: Querying, AWS Germany - prod-eu-west-2: Querying, AWS India - prod-ap-south-1: Querying, AWS Singapore - prod-ap-southeast-1: Querying, AWS Sweden - prod-eu-north-0: Querying, AWS US East - prod-us-east-0: Querying, AWS US West - prod-us-west-0: Querying, Azure Netherlands - prod-eu-west-3: Querying, Azure US Central - us-central2: Querying, GCP Australia - prod-au-southeast-0: Querying, GCP Belgium - prod-eu-west-0: Querying, GCP Brazil - prod-sa-east-0: Querying, GCP India - prod-ap-south-0: Querying, GCP Singapore - prod-ap-southeast-0: Querying, GCP UK - prod-gb-south-0: Querying, GCP US Central - prod-us-central-0: Querying).