This morning we verified that telemetry services are stable and accurately reflecting sensor status. Alerting has been restored to expected behaviors and functionality is fully restored.
As of 3:29 pm EDT today, issues related to false alerting around sensor offline telemetry status have been resolved across all sensor core versions.
We are monitoring performance tonight ahead of adjusting alerting settings tomorrow, Aug 10 at 9:30 am EDT
The issue driving the errant sensor offline notices have been isolated, and while there was no impact to log ingestion today’s incident caused distraction and noise for customers.
New alerts are no longer being processed through our notification platform. Tomorrow’s originally scheduled maintenance window has been cancelled.
We are actively investigating a problem in the sensor service that is causing sensor offline notifications to be sent to customers even when the sensors are still online.
We have temporarily paused sensor offline alerts as we investigate the cause and work toward delivering a fix as soon as possible.