HydraIssues

Runbook: no triage procedure for "node is offline — powered off, or powered on with a dead agent?"
open unclassified Project: hydra Reporter: 11 Jul 2026 09:00

Description

We just did this by hand and it worked well, but no runbook describes it. The cross-reference exists in code and is rendered as columns in hydraneck's pkg/api/templates/venue.html:159 (Agent | Network | Router | Diagnosis), but no doc teaches a human to read it, interpret a disagreement, or get from a hydracluster "offline" alert to a hydraneck venue page.

The procedure that should be written down

  1. hydracluster says the node is offline (GET /api/v1/nodes/{id}). That only means the agent has not heartbeated in >2 min (30s heartbeat, 2-min offline timeout — hydracluster/pkg/store/store.go:392-402).
  2. Cross-reference hydraneck's venue view for the node's venue. hydracluster proxies it at GET /admin/venues/{name}/network and accepts the hydracluster admin bearer token, which is the convenient path.
  3. Read the diagnosis (hydraneck/pkg/scanner/scanner.go:334-346):
    • agent-down — the router still sees the machine, the agent is silent. The box is probably up and only the agent is dead. On MikroTik this is a bound DHCP lease, not an ICMP probe, so it is suggestive rather than conclusive.
    • offline — agent silent and the router does not see it. Likely genuinely powered off/unplugged. Note this bucket also silently swallows "no router client matched" and "the router itself was unreachable", so check Routers[].State before believing it.
    • network-issue — agent is fine, router does not see it.
  4. Only after that does it make sense to reach for hydracluster exec — which is useless in the agent-down case, because exec/shell are agent-pull and require the very agent that is dead.

Where it belongs

Probably a new cross-service runbook, referenced from hydracluster/docs/runbooks/body-recovery.md (the existing offline-triage entry point) and from hydraneck/docs/runbooks/runbook.md.

The closest existing text, hydraneck/docs/runbooks/runbook.md:179-185 ("Node stuck in wrong correlation state"), names both halves of the question but gives no procedure and no answer for what to do when they disagree.