Description
On 2026-07-10, a transient DNS outage on turbo-pancake-76 (node-adf19775, Cloud Seven production head) took down WireGuard and connectivity to hydracluster/releases.experiencenet.com. The hydraheadflatscreen supervisor crash-looped relaunching the Qt kiosk app every ~30s for about an hour, then the process (and its binary on disk at ~/.hydranode/bin/hydraheadflatscreen) disappeared entirely and never respawned. hydranode (the base exec-channel agent) stayed up and reported the node as "online" the whole time, so hydracluster's node list never flagged anything wrong.
The outage went completely undetected for 13 days (2026-07-10 to 2026-07-23), discovered only because a user noticed a strange/stuck HydraExperienceNet client on the physical screen at Cloud Seven. During that window: no local API on :9740, no auto-update, no monitoring signal distinguishing this from a healthy idle head, and node.service_versions silently dropped the hydraheadflatscreen entry (visible in hindsight, but nothing alerted on it).
Fixed by POST /api/v1/nodes/{id}/reprovision, which re-ran the enrollment recipe and reinstalled the missing binary.
Ask: add monitoring/alerting for this failure mode. Candidates:
- Alert when a node with the hydraheadflatscreen role is missing its service_versions entry for that role (this alone would have caught it immediately).
- Alert when a head's local API (:9740) stops responding to periodic health polls, distinct from hydranode/exec-channel liveness.
- Alert on repeated kiosk relaunch (crash-loop) log pattern within a short window, before the process disappears entirely.
- Consider whether hydranode itself should periodically verify the hydraheadflatscreen binary exists at its configured path and self-heal (re-trigger provisioning) rather than relying on someone noticing a broken screen.