Sporadic occurrences of Grafana Cloud instances throwing 401 errors when querying a Grafana Cloud Prometheus data source

Incident Report for Grafana Cloud

Resolved

A fix for this issue has been rolled out to all of Grafana Cloud. This incident is now resolved.
Posted Jul 21, 2022 - 18:11 UTC

Identified

The issue has been identified and a fix is being implemented.
Posted Jul 19, 2022 - 16:14 UTC

Investigating

We've received several reports from customers experiencing 401 errors for Prometheus-backed dashboards. Engineering is engaged and actively investigating the issue.

This issue happens very rarely. When the issue occurs, it causes all Prometheus queries to fail until the instance is restarted.

We have determined two possible mitigation paths. Individuals can force a restart of their instance by adding or removing a plugin. If this is not an option, please submit a Support Ticket to have your affected instance restarted.
Posted Jul 15, 2022 - 19:31 UTC
This incident affected: Grafana (AWS US East (OH) - prod-us-east-2, Azure US Central - us-central2, Azure Netherlands - prod-eu-west-3, GCP Australia - prod-au-southeast-0, GCP US Central - prod-us-central-0).