Build the Linux/container peer of the Apple and Unreal pipelines, per the design accumulated in #502. Two new services, Apple naming convention.
Three axes (from #502)
Watchers keyed by VCS (hydragitwatcher = git, hydraperforcewatcher = perforce); pipelines keyed by platform (this = linux/container); builder managers keyed by platform. The SCM-agnostic build-notify {watch_name, scm_type, scm_ref, build_url} is the seam. hydragitwatcher FEEDS this pipeline; it is not part of it.
hydralinuxpipeline (control; peer of hydraapplepipeline)
- Receives the SCM-agnostic build-notify from a VCS watcher (hydragitwatcher for git).
- Tracks the build with its OWN job model (like hydraunrealengine-server: queued -> building -> pushed -> deploying -> live | failed), YAML store, bounded history.
- Asks hydrapipelinerunnerlinux for a builder and the resulting image.
- On a successful image, launches or updates the target scale via hydracluster's GENERIC op-relay (#497) using a scoped op-token, and sets user.hydra.* labels.
- Exposes GET /api/v1/builds and a dashboard (hydrapipeline-compatible read convention), SSE status via hydramonitor.
- Standard shared skeleton: hydraapi (health), hydraauth, hydraserve, hydrarelease/pkg/updater.
hydrapipelinerunnerlinux (builder manager; peer of hydrapipelinerunnerapple)
- SERVER-SIDE ONLY. Unlike the Apple runner-manager it has NO persistent on-node agent, because Linux builders are ephemeral scales spun and killed on demand. Lean enough to run as an always-on scale on a Pi (a few MB, I/O bound).
- Reads the hydraskin fleet overview via hydracluster inventory. On a build request, evaluates arch + spare capacity and NEVER starves a production node (the Pi OOM guardrail is a hard rule).
- Provisions an ephemeral builder:
- Tier 1: a rootless buildkit scale on an existing hydraskin node that has room and the right arch.
- Tier 2: an on-demand hcloud node (a cax for native arm64) via hcloud API + hydraskin install, used then destroyed.
- Dispatches the build and tears the builder down after.
Builder recipe (proven, #502)
Rootless buildkit in an UNPRIVILEGED container that DHCPs (never a manual static IP; that gave flaky egress). buildkitd --oci-worker-no-process-sandbox (skip rootlesskit, which the double user-namespace breaks). buildctl builds from the pushed context; registry auth in the builder; push to scaleregistry. Registry-backed layer cache (buildctl --export-cache/--import-cache to scaleregistry) so ephemeral builders warm-start. Build NATIVE per target arch (arm64 native on a cax; do NOT emulate arm64 for real builds, it is impractically slow).
Scales stay their own vertical
Delivery is scaleregistry + hydraskin scale launch. Do NOT route container builds through HydraRelease or HydraExperienceLibrary; those are for experiences. Keep scales separate.
Acceptance
- hydralinuxpipeline: receives build-notify, tracks builds, launches/updates a scale on success, exposes /api/v1/builds + dashboard + SSE.
- hydrapipelinerunnerlinux: places builders by arch + capacity respecting production headroom, spins ephemeral builders (scale Tier 1 or node Tier 2), builds via rootless buildkit with registry cache, tears down; runs lean on a Pi.
- rogue flows end to end: git push -> hydragitwatcher -> hydralinuxpipeline -> hydrapipelinerunnerlinux builds -> scale deployed and serving. (The live run is a staged deploy, separate from code delivery.)
Cross-reference: #502 (design), #492 (git-push-to-deploy), #497 (control-plane generic op-relay), #406 (hydraskin), #496 (special-scale config).