HydraIssues

RESOLVED — and the original diagnosis in this ticket was wrong

Correction: this ticket claimed "all binary downloads 302 to mirror-a and 404". That is
not what was happening. The release system was serving binaries fine — 796,874 successful
downloads through mirror-a since 15 April. The original report constructed test URLs from
the version string in latest.json, which is published with the v stripped, while the
real auto-updater re-adds it (hydranode pkg/updater/updater.go:144). So the fleet was
never affected and the reported symptom was an artefact of how the URLs were built by hand.

What was broken was a set of URL shapes used by humans, docs and provisioning recipes.
Two distinct defects, both now fixed and deployed.

Defect A — version-string contract mismatch (latent since forever)

Artifacts are stored under the raw git tag (.../production/v1.10.33/...), but
handleFinalize (handlers_publish.go:171) does strings.TrimPrefix(version, "v") before
publishing the manifest, so latest.json advertises 1.10.33. Concatenating the advertised
version into a download URL therefore yields a path that has never existed.

This was never a mirror regression — it predates hydramirror entirely, and was masked
because every compiled-in updater adds the v back.

Defect B — routes lost at the hydramirror cutover (the real regression)

Commit e36758d (2026-02-23) replaced the static http.FileServer with three explicit
routes, removing the latest symlink resolution and the channel-less manifest path:

The three recipes were double-broken — the dead manifest route gave an empty version and
they exited 1 at "Failed to fetch latest version"; even fixed, they would then have hit
Defect A. This is what "flaky since we introduced hydramirror" referred to.

Fixes deployed

hydrarelease (7507d8e, live as v1.17.7-1-g7507d8e) — builds on dcb0d86, which had
resolved the latest alias but could not work alone: GetLatest returns the v-less version,
so the redirect still pointed nowhere. Normalising the version is what makes it take effect.

• accept both version spellings in handleFileRedirect
• resolve the latest alias to a real path; unknown project 404s instead of redirecting to
  a literal latest directory
• re-register the channel-less /{project}/latest.json, defaulting to production
• first tests in the package, covering all three shapes

Deliberately not fixed by changing what latest.json reports — the fleet's updaters
already add the v, so that would have broken all 796k working downloads.

hydracluster (ae60920) — the three Linux recipes now use the channel-qualified
manifest and a v-prefixed download URL, matching the macOS recipes in the same repo which
already had both halves right. scripts/pi-provision.sh had the version half of the same
bug.

Verified live after deploy

Nothing was lost

mirror-a holds 2,994 files / 30 GB including every version named here. No republishing was
needed.

Recurrence guard (not yet implemented)

The blind spot is that CI asserts the manifest and never downloads the artifact — which is
why this survived five months. Add a post-publish check that downloads through the
redirect using the v-less form and verifies bytes against SHA256SUMS. See the separate
issue for the mirror-a redundancy gap found during this investigation.

Unrelated finding

hydrabooks has no published releases at all (404 even on the channel-qualified manifest),
so its recipe will still fail — for a different reason than this ticket.