HydraIssues

Proposal

Rename the release project hydrahead to hydraheadflatscreen, so the release-channel name matches the repo (cederikdotcom/hydraheadflatscreen) and the binary it actually publishes.

This is a decision document. Nothing has been changed — no code, no CI, no DNS, no hosts.

Recommendation: yes, but as a dual-publish migration, never as a rename. The release server has no alias mechanism, and there are seven confirmed consumers of the old name including three live venue Macs.

Current state (verified 2026-08-05)

.github/workflows/release.yml builds hydraheadflatscreen-* binaries, then cps them to hydrahead-* names with the comment "Updater expects binaries named hydrahead-{os}-{arch} (project name from updater config)", and publishes everything under PROJECT="hydrahead" (line 168). A second block re-publishes the Windows binary under PROJECT="hydraheadwindows" (line 189).

What is actually published under hydrahead/production/v2.1.0

Confirmed by fetching each path through releases.experiencenet.com and following the 302 (see the audit trap below):

SHA256SUMS confirms the pairs are the same bytes under two names (darwin-arm64 32a210b1…, linux-amd64 300898c5…, windows 0d97197d… each appear twice).

The alias set is already drifting. The linux/arm64 build added 2026-08-04 for the Pi fleet (commit 315e859, "Build linux/arm64 so this can run on the Pi fleet") produced hydraheadflatscreen-linux-arm64 but no hydrahead-linux-arm64 copy — the workflow's cp lines only alias windows-amd64, linux-amd64 and darwin-arm64. Any consumer resolving the hydrahead project on linux/arm64 gets a 404 right now. That is the maintenance cost of a name that matches nothing, and it broke within one release of being introduced.

hydraheadwindows/production/v2.1.0 holds a single artifact, hydraheadwindows-windows-amd64.exe (11,836,416 B, sha 0d97197d…) — byte-identical to the flatscreen Windows binary.

Audit trap: two ways to read this wrong

1. /var/www/releases/ on the release box is stale. /var/www/releases/hydrahead/production/ stops at v0.1.2, last written 2026-02-23, with a latest -> v0.1.2 symlink and a four-platform build shape CI no longer produces. It is a pre-mirror-cutover leftover. Live artifacts are on bxl1.hydramirror.experiencenet.com; releases.experiencenet.com 302s to it. Reading /var/www says this project died at v0.1.2 — it is serving v2.1.0.
2. A 302 is not proof of existence. handleFileRedirect (hydrarelease/internal/api/server.go:174) redirects to the mirror without validating the project, so a nonexistent project returns a healthy-looking 302 that 404s at the mirror. Always check with curl -L.

Why this is not cosmetic: it has already caused an outage

A Hetzner box named hydrahead ran hydrahead.service executing hydrahead serve. It was crash-looping with Error: unknown command "serve" and a systemd restart counter of 24,866. The box was retired 2026-08-04; its state (heads.yaml plus three head definitions — webstream-bxl1-01, wobbly-llama-92, test-kiosk-01) is archived at ~/backups/hydrahead-20260804/hydrahead-state.tar.gz.

Root cause, confirmed against source: the binary published under project hydrahead is the flatscreen kiosk client. Its complete command set is install, uninstall, run, diagnostics, version, update, check-update, stream-start, stream-stop. There is no serve. A server-shaped box auto-updated itself into a client binary and could never start again.

It compounds: the binary's own updater restarts the service named hydraheadflatscreen on unix, not hydrahead — so even a successful self-update would not have restarted that unit. The stale name broke both what was installed and which unit was restarted.

The failure stayed invisible because nothing tied the release-channel name back to a repo or a binary. That is the whole argument: a release project name is load-bearing — it is what a machine resolves a binary from. A name that maps to a different binary than it claims is a latent outage.

Who consumes the hydrahead / hydraheadwindows release projects

Seven confirmed consumers. Each would break on a bare rename.

1. The flatscreen binary's own auto-updater. pkg/client/client.go:90 starts updater.NewProductionUpdater("hydrahead", c.Version) on first tick; the CLI update / check-update commands do the same (internal/cli/root.go:46, :85). hydrarelease/pkg/updater/updater.go composes downloads as <project>-<GOOS>-<GOARCH> under /<project>/<channel>/v<version>/ — i.e. hydrahead-darwin-arm64. This is exactly why the cp aliases exist in CI. Highest impact: every deployed kiosk polls through this path.

2. Three live venue Macs, all role hydraheadflatscreen, all macOS arm64:

They report v2.1.0 — the exact version hydrahead/production/latest.json advertises — proving they update through the hydrahead project via consumer (1). These are the devices a careless rename strands.

Across all 20 registered cluster nodes the head roles present are hydraheadflatscreen ×3 and hydraheadipad ×1. No node holds role hydrahead or hydraheadwindows.

3. hydracluster's version poller. hydracluster/pkg/versions/poller.go:21,23 lists hydrahead and hydraheadwindows in pollableRoles — "service roles that have release binaries on the releases server" — and fetches latest.json per project on a timer. Note it does not list hydraheadflatscreen, so the cluster is polling two names no node reports while not polling the one every node does report. Up-to-date indication for these heads cannot be correct today.

4. hydranode's role-derived updater, Windows path. hydranode/pkg/body/service_windows.go:138-146 builds https://releases.experiencenet.com/<role>/production/v<ver>/<role>-windows-amd64.exe straight from the role name, with taskNameOverrides{"hydraheadwindows": "HydraHeadWindows"} at :148-150. A node with role hydraheadwindows resolves correctly against the existing project. Renaming it breaks Windows kiosk auto-provisioning.

5. The Windows install shape in this very repo. internal/cli/install_windows.go:13 — const installDir = C:\hydraheadwindows; installs hydraheadwindows.exe (:22); task XML hydraheadwindows-task.xml (:69); pkg/client/selfinstall_windows.go:52 registers a task running run --config C:\hydraheadwindows\config.yaml; internal/cli/root.go:135,154 falls back to ~/.hydraheadwindows/. The hydraheadwindows name is not a vestigial alias — it is the live on-disk identity of every Windows install.

6. hydramirror retention. hydramirror prune --project hydrahead is a documented operator command (hydramirror/docs/runbooks/runbook.md:101), and prune keys retention on releases/<project>/<channel>/ (internal/prune/plan.go:40-44), keeping newest N per project/channel. A test comment records "hydrahead has 81 versions".

7. hydraissue's own project list. hydraissue/internal/config/config.go:41,43 and config.example.yaml:16,18 list hydrahead and hydraheadwindows as tracker projects.

A broken would-be consumer worth knowing about

hydranode's non-Windows path derives the project from the role too: pkg/body/service.go:214-225 calls fetchLatestVersion(role) against /<role>/production/latest.json. For role hydraheadflatscreen that 404s, so the version falls back to the literal string latest and the URL is then built with v%s → .../hydraheadflatscreen/production/**vlatest**/hydraheadflatscreen-linux-<arch>. Three independent defects: the project doesn't exist, the version becomes vlatest, and the path is hardcoded -linux- even on macOS. updateService (service.go:141-171) additionally writes to /usr/local/bin and calls systemctl, neither of which applies on a Mac. Triggered from pkg/body/body.go:250 whenever the cluster sets UpdateServices.

Net effect today: the hydracluster "update services" action is a silent no-op for every flatscreen head. This drives the sequencing below.

Can the release server alias or redirect between project names?

No. A rename means republishing.

hydrarelease registers GET /{project}/{channel}/{version}/{file} → handleFileRedirect (internal/api/server.go:174), which 302s to <MirrorURL>/api/v1/files/releases/<project>/<channel>/<version>/<file> without validating the project. The only aliasing implemented is:

• version latest, resolved server-side via GetLatest (server.go:181-191);
• normalizeVersion accepting both 1.2.3 and v1.2.3 (server.go:156-170).

Neither crosses project names. latest.json (handleLatestJSON) requires a real release record and 404s otherwise, so a new project name cannot advertise a version without an actual publish. The old /var/www symlink trick is dead — the static file server was removed at the mirror cutover.

There is no rename or alias endpoint in the publish API; validatePublishParams (internal/api/handlers_publish.go:19-21) merely regex-validates the name. The release server itself needs no change for this migration — it has no hardcoded project names.

Good news: dual-publish is cheap and already proven — the workflow does exactly that for hydraheadwindows today. Cost is storage: retention keeps newest N per project, so a second name means a second independent retention window for the same bytes.

Proposed migration

Phase 0 — Fix hydranode first (blocking)

Publishing hydraheadflatscreen as a real project flips hydranode's role-derived updater from "404, harmless no-op" to "downloads a linux binary onto a Mac and tries to systemctl it". That is worse than the current breakage. Fix or guard serviceDownloadURL before Phase 1 lands. Related: #436.

Phase 1 — Dual-publish, no client change

Add a third PROJECT="hydraheadflatscreen" publish loop to release.yml uploading the hydraheadflatscreen-* artifacts plus finalize, keeping the hydrahead loop exactly as-is. While in there, add the missing cp for hydrahead-linux-arm64 so the old name stops drifting.

Verify: /hydraheadflatscreen/production/latest.json returns 200 with the right version, and all four artifacts return 200 after following the redirect (curl -L, not a bare 302). Backfill v2.1.0 and ideally v2.0.76 under the new project so a rollback target exists.

Add hydraheadflatscreen to pollableRoles in hydracluster at this point — it is additive and fixes the polling gap immediately.

Phase 2 — Flip the client's identity

Change the updater project in pkg/client/client.go:90, internal/cli/root.go:46/:85, the manual-download hint at root.go:53, diagnostics.go:86, and the cobra Use: — then release under both projects.

Ordering rule, non-negotiable: a device only moves to the new project after it has already installed a build that knows the new name. The old project must keep publishing for as long as any device might still be on an older build. Never flip and retire in the same release.

Phase 3 — Drain

Watch service_versions in hydracluster until all flatscreen nodes report the cutover version or later. Devices auto-update every 6h, so a healthy device converges within hours. turbo-pancake-76 is offline at v2.0.76 — it must be brought back and updated before anything is retired, or explicitly written off in writing. An offline device never converges, and shipping the cutover while it is dark guarantees it comes back pointing at a name that may no longer be published.

Phase 4 — Retire the old name

Stop publishing new versions to hydrahead after a defined window — suggest ≥2 releases and ≥30 days past the last device reporting the new name. Then drop it from pollableRoles and the hydraissue project list.

Do not delete existing artifacts. That destroys the rollback path for any device pinned to an old version, and hydramirror retention will age them out on its own schedule anyway.

What else carries the wrong name

The hydraheadwindows alias — keep it

It is byte-identical to the flatscreen Windows binary, and it is not vestigial. It is the live on-disk identity of every Windows install produced by this repo (C:\hydraheadwindows, hydraheadwindows.exe, task HydraHeadWindows, config dir ~/.hydraheadwindows), and it has a working consumer path through hydranode's role-derived updater.

No node registered in hydracluster holds that role, and the current binary on Windows self-updates from hydrahead rather than hydraheadwindows — but do not retire it on that basis. Zero registered nodes is not zero consumers: an unregistered Windows kiosk or a manual install is invisible to every check available here. Assuming a channel was unused is exactly what let a 24,866-restart crash loop persist unnoticed.

Retire it only after mirror access logs show no downloads across a full update cycle. It is independent of this rename and should be decided on its own schedule — renaming hydrahead neither helps nor hinders it.

Option: do nothing

Cost. The cp alias set must be maintained in release.yml forever, and it has already silently drifted once — hydrahead-linux-arm64 is missing at v2.1.0, a real 404 for the Pi fleet that build was added for. The release server keeps a project matching no repo, no command directory and no shipped filename. hydracluster polls two names no node reports and misses the one they all do. hydranode's role-derived updater can never work for this role. And nothing prevents a future hydrahead-named box or unit from resurrecting the failure just cleaned up.

Benefit. Zero risk to three live venue devices, zero work. The naming is confusing but currently functional — the alias copies do resolve for the platforms they cover.

Assessment. The real cost of doing nothing is not confusion; it is a manually-maintained alias set that has already broken once without anyone noticing — the same class of failure as the outage. Phase 1 alone (dual-publish + the missing cp + adding the role to pollableRoles) removes most of that risk for very little work, is purely additive, and is reversible. Phases 2–4 can be decided later on their own merits. If only one thing is approved, approve Phase 0 + Phase 1.

What could not be verified

Stated plainly, because assuming "unused" is what caused the outage.

• No download-level logging exists on anything reachable. hydrarelease's journal (6,475 lines over 14 days) contains only publish events and TLS handshake errors — zero download or redirect records — and there is no nginx on the release box. Real download logs would live on bxl1.hydramirror.experiencenet.com, outside the access available here. The consumer list above is derived from source code and cluster state only, never from observed traffic. It is complete with respect to the repos on this machine and the 20 registered cluster nodes, and may be incomplete with respect to reality.
• hydraheadwindows cannot be proven unused, for the same reason. Treat it as live.
• The v0.1.x-era consumers of hydrahead are unidentified. The stale /var/www copy shows a four-platform shape (including darwin-amd64 and linux-arm64) that today's CI does not produce. What published those, and whether anything still polls that shape, is unknown.
• Anything outside this machine's repos and the registered cluster nodes — manual installs, unregistered devices, external scripts — is not covered.