Data collection downtime
Resolved
Jun 14, 2024 at 9:08pm UTC
The incident has been resolved.
We will implement further systems to guard against this type of database misconfiguration, and are already working to improve the coverage of our monitoring systems to ensure this issue does not reoccur.
Affected services
Created
Jun 14, 2024 at 8:32pm UTC
The data collection service was offline between approx. 18:10 and 20:50 (BST) on June 14th. This was caused by a misconfiguration in a critical database, and subsequent failures in our monitoring suite.
While data was not being collected, the status quo was maintained by Micromanager installs for the first 90 minutes, they then automatically reverted to their fallback functionality when the incident was recognised.
Service was restored by 21:20, but we are unable to backfill the missing data that was not collected during this time period.
Affected services