HydraIssues

Summary

The hypothesis under evaluation was: "SSH would be a faster/better channel than hydracluster's exec queue, especially now that WireGuard tunnels are up."

Conclusion: the evidence does not support it. The ~30s exec round trip that motivated the question is not a property of the exec design — it is a stale-agent artifact that was already fixed in hydranode v1.10.32 and simply hasn't finished rolling out. On an up-to-date agent the same round trip measures 2.2s. Meanwhile, SSH-over-WireGuard would reach only 4 of 29 online nodes today, and those 4 are precisely the Windows/macOS machines where SSH is least available.

Recommendation: finish the agent rollout, keep exec as the universal control plane, and do not build an SSH path. Details below.

Evidence 1 — Measured latency, and where it actually comes from

Live measurements against the production fleet (echo / uptime, admin POST to result-in-hand):

The cause is a single ticker. internal/cli/body/run.go:74 sets the heartbeat Interval: 30 * time.Second. Historically exec was drained one command per heartbeat, so wait time was uniform over 0–30 s — exactly the 12.8 s / 25.5 s spread seen above, and exactly the "~30s" that prompted this investigation.

hydranode commit 94ea2f8 ("Drain exec queue on 3s poll instead of one-per-30s heartbeat"), first released in v1.10.32, added a dedicated goroutine in pkg/body/body.go that drains the entire queue every 3 s independent of the heartbeat. That is the 2.2 s figure.

The rollout is incomplete: 17 of 29 online nodes still run v1.10.21 or older. Online version spread: v1.10.33 x12, v1.10.21 x16, v1.10.20 x1. The runbook (docs/runbooks/runbook.md, "Remote Command Execution") still documents the old "up to 30 seconds" behaviour and should be corrected too.

This reframes the whole question. Exec latency is a tunable constant, not an architectural floor. It was already tuned by 14x. Choosing a new transport to fix a number that a completed apt upgrade already fixes would be solving the wrong problem.

Evidence 2 — WireGuard coverage in reality

Queried live from /api/v1/nodes:

• 43 nodes total, 29 online
• 9 nodes have a wireguard_ip at all
• Only 4 of those 9 are online → SSH-over-WG could reach ~14% of the online fleet
• WG peers by OS: 5 Windows, 3 macOS, 1 iOS — zero Linux

The 4 online WG nodes are cosmic-pretzel-98 (Windows), fluffy-dumpling-87 (Windows), chunky-turnip-23 (Windows), cheeky-cactus-86 (macOS). Every one carries the hydraguard-air role; they are the NAT'd render/head nodes.

There is an inversion worth naming: WireGuard covers exactly the machines where SSH is hardest (Windows/macOS, no sshd, no key management), and does not cover the 24 online Ubuntu servers where SSH would be trivial — because those have public IPs and never needed a tunnel. So the WireGuard buildout does not actually unlock SSH for the fleet in the way the hypothesis assumed.

Nodes on private LANs with no WG remain flatly unreachable by SSH: pi-node-001 (192.168.68.58), pi-node-001-b / pi-node-003-nvme / pi-node-004-nvme (10.0.0.x), cederikmini (192.168.68.55), dizzy-noodle-87 (10.110.15.188). The exec channel reaches all of these today because the agent dials outbound.

Also relevant: hydraguard is hub-and-spoke, not full mesh — README.md states all inter-site traffic routes through the hub, spokes carry AllowedIPs = 10.10.0.0/16, 10.0.0.0/8 and the hub holds per-peer /32 routes. The hydracluster server is not itself a mesh peer. Any SSH path would first require hydracluster to join the mesh or proxy through the hydraguard hub as a jump host — additional moving parts on the critical management path.

Evidence 3 — What SSH would genuinely add

Honest accounting. Two of these are real:

• Streaming output — REAL GAP. pkg/body/exec.go buffers stdout/stderr into bytes.Buffer and returns them only on completion. Nothing streams. A 10-minute command produces no output until it ends.
• File transfer (scp/rsync) — REAL GAP. Today you base64 a payload through a command string. Genuinely awkward for anything large.
• Long-running commands — partial. executeCommand defaults to a 30 s timeout, caller-overridable via req.Timeout. Workable, but there is no detach/reattach for multi-hour jobs.
• Interactivity — NOT a gap. hydracluster already has an interactive PTY over WebSocket: GET /api/v1/body/shell with a 3 s /api/v1/body/shell/check poll (pkg/body/shell.go), surfaced as hydracluster exec --shell and /admin/shell/{id}. The agent dials outbound, so it works behind NAT. This is the single most-cited SSH advantage and it is already built.
• Port forwarding — not a gap in practice. WireGuard already provides this for the nodes that have it; SSH would be redundant.
• Lower latency — not a gap after rollout. 2.2 s vs SSH's ~0.3 s is a real but small delta for an operator-driven control plane.

Evidence 4 — What exec gives that SSH does not

This is the side most at risk of being underweighted:

1. Outbound-only, universal NAT/firewall traversal. The agent polls out; nothing needs inbound reachability. This is why exec works on every private-LAN node listed above. SSH inverts this and requires either a public IP or a tunnel.
2. Bootstrapping — the decisive argument. hydraguard-air is provisioned through this very channel: recipes/hydraguard-air-{linux,macos,windows}.yaml write {{.WireGuardConfig}} and activate the tunnel via exec steps. WireGuard is installed by exec. A design where losing WireGuard also loses your management path cannot repair itself — you would be locked out of exactly the node you need to fix, with a site visit as the only recovery. Any SSH path must stay strictly subordinate to exec for this reason alone.
3. Auth reuse. Exec rides the existing node-token model (requireNodeToken). SSH means generating, distributing, rotating and revoking keys across Windows and macOS endpoints — a new secret-management surface. Note: no recipe anywhere provisions sshd, keys, or authorized_keys (grepped recipes/ — zero hits). This is not "turn on a flag"; it is a subsystem that does not exist yet.
4. Central audit trail. Every command traverses the server, is queued, attributed and inspectable (GET /api/v1/nodes/{id}/exec/queue, plus the queue-clear and per-exec-cancel endpoints). SSH sessions bypass the server entirely — no central record. Issue #184 (exec attribution) is actively improving this; SSH would move in the opposite direction.
5. Windows support. pkg/body/exec.go invokes PowerShell by absolute path with -NoProfile -NonInteractive, plus ensureWindowsPath() which repairs a corrupted system PATH before every command — hard-won hardening against real incidents. Windows is a large share of the render fleet and has no sshd today.
6. Fleet-wide uniformity. One transport, one auth model, one audit path, one set of runbooks, across Linux/Windows/macOS/iOS.

Options and honest tradeoffs

A. Finish the v1.10.33 rollout (17 stale nodes) — RECOMMENDED, do first.
Cost: near zero; the code is written, released and proven in production. Effect: 30 s → 2.2 s fleet-wide, i.e. the entire reported problem. Also update the runbook's stale "up to 30 seconds" wording. Risk: the auto-updater evidently is not converging some nodes — worth a look at why v1.10.21 is pinned on 16 machines, since that is a fleet-management bug in its own right.

B. Wake-on-command push (long-poll, or reuse the existing shell WebSocket) — optional, later.
Cuts 2.2 s to sub-second while keeping one transport, one auth model, one audit trail. Cost: small-to-moderate, server + agent. Only worth doing if sub-second turns out to matter after (A); 2.2 s is likely fine for operator-driven work.

C. Close the two real gaps in-channel: streamed output and file transfer.
Chunked incremental result posting, and a proper push/pull file endpoint. Cost: moderate. This addresses what SSH would genuinely have bought, without a second transport, and benefits all nodes rather than 14% of them.

D. SSH as an opportunistic fast path where a WG address exists — NOT recommended.
Cost: sshd + key provisioning/rotation on Windows and macOS (does not exist today), hydracluster joining the WG mesh or a hub jump-host, a second auth path, a second audit path, fallback logic, and doubled runbooks. Benefit: reaches 4 of 29 online nodes, on the two OSes where SSH is least natural, to save ~2 s on a path that already has an interactive PTY. Poor return, and it dilutes the single-control-plane property that makes the fleet debuggable.

Recommendation

Do (A) now; treat (C) as the real backlog item; skip (D).

"Faster" was the wrong frame. The exec channel's latency was dominated by a poll interval — a tunable constant that has already been tuned by 14x and merely needs deploying. The genuine limitations of the exec channel are streaming output and file transfer, neither of which requires changing transport to fix, and both of which are better fixed in-channel so they benefit the whole fleet rather than the 14% with a tunnel.

Keep exec as the universal control plane. It is the only path that reaches every node regardless of NAT, OS, or WireGuard state — and critically, it is the path that installs and repairs WireGuard itself. Making the management plane depend on WireGuard would be a bootstrapping trap with site-visit recovery. If SSH is ever added, it must be a strictly optional convenience layer for the public-IP Linux servers, never the primary path, and never the only way to reach a node.

Evidence gathered read-only from hydracluster, hydranode and hydraguard sources plus the live /api/v1/nodes API. Related: #171 (the drain fix, done), #184 (exec attribution), #327 / #326 (exec reliability).

Addendum 2026-08-03: does upgrading hydranode alone close this?

Question raised while planning a migration: if the fleet is being upgraded anyway, does that resolve the exec complaints? It resolves the latency finding above and nothing else. Verified read-only against the current hydranode and hydracluster working trees.

Confirmed still true

• Latest hydranode tag is v1.10.33. The drain goroutine is present at pkg/body/body.go:108-115: a dedicated 3s ticker calling drainExecQueue(), independent of the heartbeat. The 2.2s figure holds, and option (A) remains correct.

Confirmed NOT fixed by an agent upgrade

These are hydracluster server side, so no hydranode version affects them:

• #327 (server-side timeout enforcement, in-flight eviction on disconnect, in-flight cancellation): no inFlight sweep / evict / expire code exists anywhere in pkg/api/. Unimplemented as of today. A hung exec is still unrecoverable without manual intervention.
• #326 (hydracluster exec unblock): no CLI surface. The only matches are the pre-existing ClearQueue comments at pkg/api/exec_store.go:115 that the issue already cites.
• #307 (Stop button unreliable): a faster drain shrinks the window but does not change the root cause, which is hydracluster clearing stream_status regardless of exec outcome.
• The two REAL GAPs named above, streaming output and file transfer, are agent side but not in any released version. pkg/body/exec.go still buffers stdout/stderr to completion. This is option (C) and it remains the real backlog item.
• #180 (Mac kiosk exec dies after Moonlight exits): status unknown. The report predates v1.10.32, and a hung subprocess would park the drain goroutine under either model. Worth retesting on a current agent before assuming it survived the change.

New finding: the rollout may not be purely automatic

Option (A) above assumes stale nodes catch up via auto-update. That assumption needs checking before the migration:

• hydranode/docs/runbooks/runbook.md:133 records that macOS installs prior to v1.10.29 have a root-owned binary at /usr/local/bin/hydranode which blocks auto-update. Those need a one-time sudo hydranode reinstall to move the binary to the user-owned path and update the plist.
• #171 separately reported Windows auto-update getting stuck and macOS service version checks failing.

The OS breakdown of the 16 nodes that were on v1.10.21 was not captured, and the evidence above indicates most of the online fleet is Linux, so this may not be what is holding them. But "finish the rollout" should be scoped as possibly needing manual intervention per platform rather than a single sweep.

Not re-measured

The version spread (v1.10.33 x12, v1.10.21 x16, v1.10.20 x1 as of 2026-07-28) was not re-checked on 2026-08-03. Someone should confirm current state before sizing the remaining rollout work.