HydraIssues

Control-plane vs node-agent split: hydraskin op API, hydracluster delegation, scoped control-plane tokens
open feature Project: hydracluster Reporter: anonymous 18 Aug 2026 08:45

Description

Context: The git-push-to-deploy work (#492) surfaced an architectural smell. Today the only way to drive a node remotely is hydracluster's arbitrary exec (run shell on a node as root). A workload that needs to launch or update a scale (the git-push watcher) therefore must hold the hydracluster admin token, which is fleet root. Arbitrary exec is fundamentally unscopeable, so the admin token is all-or-nothing.

Target architecture (control plane vs node agent, like the Kubernetes API-server and scheduler versus the kubelet):

  • hydraskin (node agent) owns all scale lifecycle on its node behind a real, structured operation API: launch, update, expose, remove, report. It is the only thing that touches a node's Incus. Operations accept the generic scale knobs from #496 (isolation unprivileged|nesting|vm, entrypoint, internal-by-default, env and secret refs, disk, resources).
  • hydracluster (control plane) does what a single node cannot: membership and enrollment, role assignment and provisioning, placement (which node), fleet inventory, transport to NAT'd nodes, and control-plane authz. It DELEGATES scale operations to the target node's hydraskin over the existing node-to-control-plane transport, carrying a structured operation, not shell.
  • Retire arbitrary exec as the deploy path. exec remains a break-glass, admin-only capability, not the mechanism workloads use.
  • Scoped, revocable control-plane tokens (RBAC): tokens are records with allowed operations, resource scope (nodes, scale-name prefix, realm or org), expiry, revocation, and audit. Deny-by-default. The admin token becomes break-glass root. A workload like the git-push watcher gets a token that can only create or update scales from the scale registry on its realm's nodes and set user.hydra.* labels, nothing else. Ideally tokens tie to an iamnim org.

Transport note: the op API must ride the SAME channel exec uses to reach NAT'd venue nodes (the node dials the control plane). Do not require inbound access to nodes. Replace the shell payload with a structured operation payload, authorized at the control plane before forwarding.

Acceptance:

  • hydraskin exposes structured scale operations (launch, update, expose, remove, report) with the #496 knobs; no shell required to deploy a scale.
  • hydracluster exposes a deploy-scale operation that authorizes against a scoped token, accepts or picks the node, and delegates to hydraskin over the existing transport.
  • Scoped, revocable, audited control-plane tokens exist; deny-by-default; the admin token is break-glass only.
  • The git-push watcher (#492) uses a scoped token, never the admin token, and never arbitrary exec.
  • Existing scales and the current exec channel keep working during migration (no big-bang break).

Cross-reference: #492 (git-push-to-deploy), #496 (declarative special-scale config), #406 (hydraskin container host role), and the iamnim CLI-auth work for the token model.