HydraIssues

Linux pipeline: deployment hardening + multi-tenant follow-ups
open unclassified Project: hydralinuxpipeline Reporter: 19 Aug 2026 21:42

Description

Follow-ups surfaced while deploying the Linux pipeline (hydralinuxpipeline + hydrapipelinerunnerlinux + hydragitwatcher) as live scales. None block the deployment; all are hardening / multi-tenant polish.

  1. HOST-PORT AUTO-ALLOCATION (hydraskin). hydraskin launch defaults the node host-port to the container port, so a second scale on one node that also wants :8080 fails to bind. Multiple projects on one node need distinct node ports. Allocate a free host port when the default is taken (hydraskin expose already has AllocatePort 30000-32767; reuse it in launch, or have the pipeline pass a distinct --host-port).

  2. SCOPE THE EXEC TOKEN (hydracluster / pipeline / runner). The deployed pipeline and runner scales hold the hydracluster ADMIN token (cluster.exec_token) in their config to drive incus/hydraskin over the exec transport. Add a cluster-side scoped exec token/allowlist that only permits hydraskin ... and the builder incus launch/exec/delete, and give the scales that instead of admin.

  3. buildctl --secret for the registry push token (hydrapipelinerunnerlinux). The push token currently reaches the ephemeral builder inline in the base64 build script (visible in the exec log). Switch to buildctl --secret or an incus file push-ed auth file.

  4. NOTIFY-ONLY SKIPS THE FETCH (hydragitwatcher). In notify-only mode the watcher still shallow-fetches the commit before notifying, which is redundant (the pipeline's runner clones itself). Skip it unless a .hydrabuild.yaml needs reading.

Related: #508.

  1. RETRY LOSES PER-JOB DEPLOY INTENT (hydralinuxpipeline). handleRetryBuild (internal/server/api.go) copies WatchName/ScmType/ScmRef/BuildURL/Arch/Project/Scale but NOT Domain/Port/HealthPath/Disks/TargetNodeID, so a retried job deploys with the global config to the build node instead of the project's intended node/domain. Observed: a v1.1.3 retry deployed rogue onto the arm builder node instead of the Pi. Copy all deploy-intent fields in retry.

  2. DEPLOY MUST BE IDEMPOTENT: launch -> update on 'already exists' (hydralinuxpipeline deploy client). The pipeline picks launch vs update from its own job history (HasLiveScale), which is empty after a restart or for a scale it did not create - so it issues hydraskin launch for a scale that already exists and fails ('already exists - use hydraskin update'). The deploy client should, on an 'already exists' failure from launch, retry as update (or hydraskin launch should upsert). This bit every deploy of an already-running scale from a fresh pipeline.