Things appear stable and we are marking this incident as resolved.
Posted Dec 10, 2019 - 23:28 UTC
Monitoring
We will leave the offending code disabled until we can fix the issue, in the meantime we will monitor the cluster and make sure it remains stable.
Posted Dec 10, 2019 - 22:32 UTC
Identified
The source of the issue is related to a planned schema update from v9 to v11, it appears during the update the table-manager component was incorrectly deleting the new table and logs were not able to saved.
The deletion code was disabled and the new table was created allowing logs to save.
The affected period appears to be 2019/12/10 21:10 UTC to 2019/12/10 22:02 UTC
Posted Dec 10, 2019 - 22:23 UTC
Investigating
We were alerted of ingester restarts in the US-CENTRAL cluster which likely have resulted in some lost logs. Things are stable and we are investigating further.
Posted Dec 10, 2019 - 22:18 UTC
This incident affected: Grafana Cloud: Loki (GCP US Central - prod-us-central-0).