Resolved
This incident has been resolved.
Posted Feb 18, 2025 - 14:06 UTC
Update
Our engineering teams have applied fixes to tackle the issues, and all services on the AWS clusters should be operational.
We will continue monitoring them.
Posted Feb 18, 2025 - 13:20 UTC
Update
Update in affected components: Prometheus Metrics on us-east-0 and services depending on this are back to operational.
Posted Feb 18, 2025 - 13:06 UTC
Monitoring
Update in affected components: Synthetics Monitoring components back to operational, services depending on Prometheus Metrics on us-east-0 still under examination.
We are continuing to monitor all the services across AWS clusters. Beware that some services may still have degraded performance until fully infrastructure stabilization.
Posted Feb 18, 2025 - 12:40 UTC
Update
We are continuing to work on a fix for this issue.
Posted Feb 18, 2025 - 12:23 UTC
Update
Update in affected components: IRM components partially recovered, Oncall services are fully operational, Incident services recovering.
Prometheus services are almost fully operational (monitoring recovery on us-east-0)
Posted Feb 18, 2025 - 12:21 UTC
Update
Update in affected components: Tempo services and asserts services have been restored, alerting services have been partially restored.
Currently monitoring all operative services.
Posted Feb 18, 2025 - 12:11 UTC
Identified
We have identified the issue, and we are bringing back to operational state most of the services including: Loki services, Pyroscope services, and AI/ML Services. We are monitoring these services.
Posted Feb 18, 2025 - 11:53 UTC
Update
Update in affected components: OTLP Endpoint and Graphite proxy for querying and ingesting are fully operational.
Posted Feb 18, 2025 - 11:24 UTC
Update
We are continuing to investigate this issue and working on reestablishing the service.
Posted Feb 18, 2025 - 10:57 UTC
Update
Update in affected components: Hosted Grafana instances (stacks) are operational.
Posted Feb 18, 2025 - 10:14 UTC
Update
Update in affected components: Grafana Cloud k6 (and legacy app.k6.io) are fully operational.
Posted Feb 18, 2025 - 10:06 UTC
Update
We are continuing to investigate this issue and determining the full impact.
Posted Feb 18, 2025 - 10:01 UTC
Update
Update in components scope: potentially all our services running on AWS may be affected.
Posted Feb 18, 2025 - 09:44 UTC
Update
Update in components scope: potentially all our services running on AWS may be affected.
Posted Feb 18, 2025 - 09:43 UTC
Update
We are continuing to investigate this issue.
Posted Feb 18, 2025 - 09:33 UTC
Investigating
We are currently experiencing an outage on our instances locate on AWS cloud due to DNS problems.
We are actively working to reestablish the service and quantify the whole impact of the issue. All our services running on this provider may be potentially affected.
Posted Feb 18, 2025 - 08:43 UTC
This incident affected: Grafana Cloud: Pyroscope (AWS Australia - prod-au-southeast-1, AWS Brazil - prod-sa-east-1, AWS Canada - prod-ca-east-0, AWS Germany - prod-eu-west-2, AWS India - prod-ap-south-1, AWS Japan - prod-ap-northeast-0, AWS Singapore - prod-ap-southeast-1, AWS Sweden - prod-eu-north-0, AWS US East - prod-us-east-0, AWS US West - prod-us-west-0), Grafana Cloud: Hosted Grafana (AWS Australia - prod-au-southeast-1, AWS Brazil - prod-sa-east-1, AWS Canada - prod-ca-east-0, AWS Germany - prod-eu-west-2, AWS Germany - prod-eu-west-4, AWS India - prod-ap-south-1, AWS Japan - prod-ap-northeast-0, AWS Singapore - prod-ap-southeast-1, AWS Sweden - prod-eu-north-0, AWS US East - prod-us-east-0, AWS US East - prod-us-east-2, AWS US West - prod-us-west-0), Grafana Cloud: Graphite (AWS Australia - prod-au-southeast-1: Querying, AWS Australia - prod-au-southeast-1: Ingestion, AWS Brazil - prod-sa-east-1: Querying, AWS Brazil - prod-sa-east-1: Ingestion, AWS Canada - prod-ca-east-0: Querying, AWS Canada - prod-ca-east-0: Ingestion, AWS Germany - prod-eu-west-2: Querying, AWS Germany - prod-eu-west-2: Ingestion, AWS India - prod-ap-south-1: Querying, AWS India - prod-ap-south-1: Ingestion, AWS Japan - prod-ap-northeast-0: Querying, AWS Japan - prod-ap-northeast-0: Ingestion, AWS Singapore - prod-ap-southeast-1: Querying, AWS Singapore - prod-ap-southeast-1: Ingestion, AWS Sweden - prod-eu-north-0: Querying, AWS Sweden - prod-eu-north-0: Ingestion, AWS US East - prod-us-east-0: Querying, AWS US East - prod-us-east-0: Ingestion, AWS US West - prod-us-west-0: Querying, AWS US West - prod-us-west-0: Ingestion), Grafana Cloud: Prometheus (AWS Australia - prod-au-southeast-1: Querying, AWS Australia - prod-au-southeast-1: Ingestion, AWS Brazil - prod-sa-east-1: Querying, AWS Brazil - prod-sa-east-1: Ingestion, AWS Canada - prod-ca-east-0: Querying, AWS Canada - prod-ca-east-0: Ingestion, AWS Germany - prod-eu-west-2: Querying, AWS Germany - prod-eu-west-2: Ingestion, AWS India - prod-ap-south-1: Querying, AWS India - prod-ap-south-1: Ingestion, AWS Japan - prod-ap-northeast-0: Querying, AWS Japan - prod-ap-northeast-0: Ingestion, AWS Singapore - prod-ap-southeast-1: Querying, AWS Singapore - prod-ap-southeast-1: Ingestion, AWS Sweden - prod-eu-north-0: Querying, AWS Sweden - prod-eu-north-0: Ingestion, AWS US East - prod-us-east-0: Ingestion, AWS US West - prod-us-west-0: Querying, AWS US West - prod-us-west-0: Ingestion), Grafana Cloud: Prometheus Rules and Alerts (AWS Australia - prod-au-southeast-1: Rule Evaluation, AWS Australia - prod-au-southeast-1: Alertmanager and Rules Configuration API, AWS Australia - prod-au-southeast-1: Alertmanager, AWS Brazil - prod-sa-east-1: Rule Evaluation, AWS Brazil - prod-sa-east-1: Alertmanager and Rules Configuration API, AWS Brazil - prod-sa-east-1: Alertmanager, AWS Canada - prod-ca-east-0: Rule Evaluation, AWS Canada - prod-ca-east-0: Alertmanager and Rules Configuration API, AWS Canada - prod-ca-east-0: Alertmanager, AWS Germany - prod-eu-west-2: Rule Evaluation, AWS Germany - prod-eu-west-2: Alertmanager and Rules Configuration API, AWS Germany - prod-eu-west-2: Alertmanager, AWS India - prod-ap-south-1: Rule Evaluation, AWS India - prod-ap-south-1: Alertmanager and Rules Configuration API, AWS Japan - prod-ap-northeast-0: Rule Evaluation, AWS Japan - prod-ap-northeast-0: Alertmanager and Rules Configuration API, AWS Singapore - prod-ap-southeast-1: Rule Evaluation, AWS Singapore - prod-ap-southeast-1: Alertmanager and Rules Configuration API, AWS Singapore - prod-ap-southeast-1: Alertmanager, AWS Sweden - prod-eu-north-0: Alertmanager and Rules Configuration API, AWS Sweden - prod-eu-north-0: Alertmanager, AWS Sweden - prod-eu-north-0: Rule Evaluation, AWS US East - prod-us-east-0: Rule Evaluation, AWS US East - prod-us-east-0: Alertmanager and Rules Configuration API, AWS US East - prod-us-east-0: Alertmanager, AWS US West - prod-us-west-0: Rule Evaluation, AWS US West - prod-us-west-0: Alertmanager and Rules Configuration API, AWS US West - prod-us-west-0: Alertmanager), Grafana Cloud: Loki (AWS Australia - prod-au-southeast-1, AWS Brazil - prod-sa-east-1, AWS Canada - prod-ca-east-0, AWS Germany - prod-eu-west-2, AWS India - prod-ap-south-1, AWS Japan - prod-ap-northeast-0, AWS Singapore - prod-ap-southeast-1, AWS Sweden - prod-eu-north-0, AWS US East - prod-us-east-0, AWS US West - prod-us-west-0), Grafana Cloud: Tempo (AWS Australia - prod-au-southeast-1: Querying, AWS Australia - prod-au-southeast-1: Ingestion, AWS Australia - prod-au-southeast-1: Metrics Generator, AWS Brazil - prod-sa-east-1: Querying, AWS Brazil - prod-sa-east-1: Ingestion, AWS Brazil - prod-sa-east-1: Metrics Generator, AWS Canada - prod-ca-east-0: Querying, AWS Canada - prod-ca-east-0: Ingestion, AWS Canada - prod-ca-east-0: Metrics Generator, AWS Germany - prod-eu-west-2: Querying, AWS Germany - prod-eu-west-2: Ingestion, AWS Germany - prod-eu-west-2: Metrics Generator, AWS India - prod-ap-south-1: Querying, AWS India - prod-ap-south-1: Ingestion, AWS India - prod-ap-south-1: Metrics Generator, AWS Japan - prod-ap-northeast-0: Querying, AWS Japan - prod-ap-northeast-0: Ingestion, AWS Japan - prod-ap-northeast-0: Metrics Generator, AWS Singapore - prod-ap-southeast-1: Querying, AWS Singapore - prod-ap-southeast-1: Ingestion, AWS Singapore - prod-ap-southeast-1: Metrics Generator, AWS Sweden - prod-eu-north-0: Querying, AWS Sweden - prod-eu-north-0: Ingestion, AWS Sweden - prod-eu-north-0: Metrics Generator, AWS US East - prod-us-east-0: Querying, AWS US East - prod-us-east-0: Ingestion, AWS US East - prod-us-east-0: Metrics Generator, AWS US West - prod-us-west-0: Querying, AWS US West - prod-us-west-0: Ingestion, AWS US West - prod-us-west-0: Metrics Generator), Grafana Cloud: Machine Learning (AWS Australia - prod-au-southeast-1, AWS Brazil - prod-sa-east-1, AWS Canada - prod-ca-east-0, AWS Germany - prod-eu-west-2, AWS India - prod-ap-south-1, AWS Japan - prod-ap-northeast-0, AWS Singapore - prod-ap-southeast-1, AWS Sweden - prod-eu-north-0, AWS US East - prod-us-east-0, AWS US West - prod-us-west-0), Grafana Cloud: Synthetic Monitoring (AWS Australia - prod-au-southeast-1: API, AWS Australia - prod-au-southeast-1: Public Probes, AWS Brazil - prod-sa-east-1: API, AWS Brazil - prod-sa-east-1: Public Probes, AWS Canada - prod-ca-east-0: API, AWS Canada - prod-ca-east-0: Public Probes, AWS Germany - prod-eu-west-2: API, AWS Germany - prod-eu-west-2: Public Probes, AWS India - prod-ap-south-1: API, AWS India - prod-ap-south-1: Public Probes, AWS Japan - prod-ap-northeast-0: API, AWS Japan - prod-ap-northeast-0: Public Probes, AWS Singapore - prod-ap-southeast-1: API, AWS Singapore - prod-ap-southeast-1: Public Probes, AWS Sweden - prod-eu-north-0: API, AWS Sweden - prod-eu-north-0: Public Probes, AWS US East - prod-us-east-0: API, AWS US East - prod-us-east-0: Public Probes, AWS US West - prod-us-west-0: API, AWS US West - prod-us-west-0: Public Probes), Grafana Cloud: OTLP Endpoint (AWS Australia - prod-au-southeast-1, AWS Brazil - prod-sa-east-1, AWS Canada - prod-ca-east-0, AWS Germany - prod-eu-west-2, AWS India - prod-ap-south-1, AWS Japan - prod-ap-northeast-0, AWS Singapore - prod-ap-southeast-1, AWS Sweden - prod-eu-north-0, AWS US East - prod-us-east-0, AWS US West - prod-us-west-0), Grafana Cloud: Faro (AWS Australia - prod-au-southeast-1, AWS Brazil - prod-sa-east-1, AWS Canada - prod-ca-east-0, AWS Germany - prod-eu-west-2, AWS India - prod-ap-south-1, AWS Japan - prod-ap-northeast-0, AWS Singapore - prod-ap-southeast-1, AWS Sweden - prod-eu-north-0, AWS US East - prod-us-east-0, AWS US West - prod-us-west-0), Grafana Cloud: Private Datasource Connect (PDC) (AWS Australia - prod-au-southeast-1, AWS Brazil - prod-sa-east-1, AWS Canada - prod-ca-east-0, AWS Germany - prod-eu-west-2, AWS India - prod-ap-south-1, AWS Japan - prod-ap-northeast-0, AWS Sweden - prod-eu-north-0, AWS US East - prod-us-east-0, AWS US West - prod-us-west-0), Grafana Cloud: Insight Logs (AWS Australia - prod-au-southeast-1, AWS Brazil - prod-sa-east-1, AWS Canada - prod-ca-east-0, AWS Germany - prod-eu-west-2, AWS Germany - prod-eu-west-4, AWS India - prod-ap-south-1, AWS Japan - prod-ap-northeast-0, AWS Singapore - prod-ap-southeast-1, AWS Sweden - prod-eu-north-0, AWS US East - prod-us-east-0, AWS US East - prod-us-east-2, AWS US West - prod-us-west-0), Grafana Cloud: Adaptive Metrics (AWS Australia - prod-au-southeast-1, AWS Brazil - prod-sa-east-1, AWS Canada - prod-ca-east-0, AWS Germany - prod-eu-west-2, AWS Germany - prod-eu-west-4, AWS India - prod-ap-south-1, AWS Japan - prod-ap-northeast-0, AWS Singapore - prod-ap-southeast-1, AWS Sweden - prod-eu-north-0, AWS US East - prod-us-east-0, AWS US East - prod-us-east-2, AWS US West - prod-us-west-0), Grafana Cloud: k6 (GCk6 App, Cloud Test Runs, Cloud Output Test Runs, Result Analysis, Test Scheduling, Private Load Zones, Cloud logs), and Grafana Cloud: Integrations, Grafana Cloud: AWS Logs, Grafana Incident, Grafana OnCall, Grafana Cloud: SLO.