worldping gateway errors

Incident Report for Grafana Cloud

Resolved

All services have continued to operate normally.

Posted Sep 12, 2017 - 17:10 UTC

Monitoring

Services have now been restored.
The root cause was due to a cascading failure triggered by the hosted-metrics outage. Due to a software bug in the probes, after ingestion to the timeseries database was restored after a few minutes the probes tried sending all of their buffered data at once instead of in chunks. The payloads the probes were trying to send required too much memory to process and lead to a crash of the gateway API services.

Posted Sep 12, 2017 - 16:19 UTC

Identified

The worldping api gateways are experiencing issues. This is affecting both querying and ingestion of new metrics from probes. Engineers are working to restore services.

Posted Sep 12, 2017 - 15:46 UTC