HydraIssues

Problem

When an exec command gets stuck (cmd.Run() blocks on the agent side), the drain goroutine is parked and all subsequent execs for that node pile up waiting. There is currently no CLI tooling to recover this — an operator must:

1. Know the raw API endpoints
2. Manually curl DELETE /api/v1/nodes/{id}/exec/queue to clear the backlog
3. Separately use the shell to find and kill the hanging subprocess on the body

Shell and exec are independent channels, so the shell remains usable even when exec is frozen. This is the recovery path but it is completely undiscoverable and requires knowing internal curl commands.

Solution

Add a hydracluster exec unblock <nodeId> CLI subcommand that:

1. Calls GET /api/v1/nodes/{id}/exec/queue to identify the in-flight command
2. Opens a shell session to the node and kills the hanging subprocess (find by the command string from the in-flight exec)
3. Calls DELETE /api/v1/nodes/{id}/exec/queue to clear the backlog
4. Confirms drain goroutine is unblocked by checking queue status

Also add standalone queue management subcommands:

• hydracluster exec queue-status <nodeId> — show in-flight and queued items
• hydracluster exec queue-clear <nodeId> — clear queued (not in-flight) items

These map directly to the existing API endpoints (GET/DELETE /api/v1/nodes/{id}/exec/queue) which are already implemented server-side but have no CLI surface.

Context

Identified during pre-event analysis 2026-05-21. The shell WebSocket (handlers_shell.go) is a separate channel from exec and stays alive when exec is frozen — this is the key insight that makes unblock possible without a hydrabody restart.