HydraIssues

hydranode/hydraheadflatscreen do not survive a physical reboot on Mac Mini kiosk heads (turbo-pancake-76), no remote recovery possible
open bug Project: hydranode Reporter: agentcodex 27 Jul 2026 12:59

Description

On turbo-pancake-76 (node-adf19775, Cloud Seven production head), the operator physically restarted the Mac Mini. After the restart, neither hydranode nor hydraheadflatscreen came back up automatically:

- hydracluster's node list shows the node as status: offline (not just hydraheadflatscreen missing service_versions like issue #400 — the base hydranode exec channel itself never checked back in).
- `hydracluster exec node-adf19775 ...` times out completely (not connection-refused) — hydranode is simply not running, so there is no remote path to fix anything. This is a step worse than the issue #400 scenario: there we could still exec in and reprovision remotely; here we cannot reach the machine at all.
- With nothing auto-launching, the operator fell back to double-clicking HydraExperienceNet.app directly in Finder. Without the `kiosk --district <d> --venue <v>` launch arguments that hydraheadflatscreen normally supplies, this opens the stock Moonlight-Qt UI (generic PC-list/settings screen) instead of the branded kiosk grid — confusing and non-functional for on-site/venue staff ("ugly version that doesn't work").

Suspected root cause: hydranode and hydraheadflatscreen are both installed as user LaunchAgents (`~/Library/LaunchAgents/com.experiencenet.{hydranode,hydraheadflatscreen}.plist`). LaunchAgents only start once a GUI user session begins — if the Mac Mini is not configured for (or fails) autologin after a restart, the machine sits at the login screen and nothing in LaunchAgents fires until a human physically logs in. Since there's no GUI session, there's also no way to remotely trigger anything (hydranode itself is what would normally let us do that). Manual recovery required physical presence + Terminal + `launchctl bootstrap gui/$(id -u) <plist>` for both services.

Ask:
1. Audit/confirm autologin is enabled and working on every Mac Mini kiosk head in the fleet (cheeky-cactus-86, peppy-dumpling-32, turbo-pancake-76, and any others), not just this one.
2. Consider moving hydranode specifically to a LaunchDaemon (root, starts at boot before any GUI login) instead of a per-user LaunchAgent, so the remote exec channel survives a reboot even if the GUI/autologin session has a problem. The Qt kiosk app still needs a GUI session so hydraheadflatscreen itself may need to stay a LaunchAgent, but recovering hydranode alone would restore remote fixability.
3. Add fleet-wide alerting on node status flipping to offline (distinct from #400's "online but service silently dead" case) so this is caught immediately instead of discovered by an operator seeing a broken kiosk.
4. Make the manual-fallback story safer: HydraExperienceNet.app launched standalone (no kiosk args) currently shows the raw upstream Moonlight-Qt UI, which is misleading for non-technical on-site staff. Consider detecting a no-args launch and showing a clear "not configured — this should be launched via hydraheadflatscreen" message instead of the generic PC-list screen.

Related: #400 (hydraheadflatscreen agent process/binary disappearing silently while hydranode stayed up) and #401 (kiosk fullscreens on macOS Main Display, no configurable target) — same fleet, same head, different failure layers.