As of 22:20 UTC, querying has been restored for both Graphite and Prometheus, and users should no longer be experiencing the 500 internal Server Error.
Posted Apr 10, 2024 - 22:26 UTC
Monitoring
As of 19:17 UTC, we have observed marked recovery and users in the `prod-us-central-0` cluster should now be able to query Graphite and Prometheus metrics without errors.
We will monitor this issue and provide updates as more information is shared.
Posted Apr 10, 2024 - 19:41 UTC
Investigating
As of 18:30 UTC, we were alerted to a high volume of errors querying metrics in both Graphite and Prometheus within the prod-us-central-0 cluster only. Users encountering this issue may experience an error stating "500 Internal Server Error: the bucket index is too old" when querying metrics.
Engineering is actively engaged and assessing possible remediation paths. We will provide updates as more information is shared.
Posted Apr 10, 2024 - 19:05 UTC
This incident affected: Grafana Cloud: Prometheus (GCP US Central - prod-us-central-0: Querying).