HydraIssues

Issues / #95
hydraperforcewatcher publishes external builds to dead URL path, bodies 404 on download
done bug Project: hydraperforcewatcher Reporter: cederik 21 Apr 2026 14:54

Description

Problem

External Perforce builds registered in hydraexperiencelibrary via POST /api/v1/builds/notify carry a build_url pointing at https://releases.experiencenet.com/builds/<project>/<cl>/Builds/<file>.zip, but hydrarelease no longer serves that path. Every body polling /api/v1/experiences/live receives this URL, tries to download, and gets HTTP 404. Experiences appear healthy in the library dashboard and in the promote/rollback API, while no fleet machine can actually install the build.

Concrete evidence from today (mercator-talks CL 91 / V0.9)

  1. hydraperforcewatcher (visitflanders agent) successfully rsync-uploaded the zip on 2026-03-30 to the release server at /var/www/releases/builds/soulmade/91/Builds/MercatorV0.9.zip (1.94 GB, sha256 0f67c044a24d13583228e40c4c5cac2a652735b920a177705168c31fe81cf002). File still intact there.
  2. Build was registered in hydraexperiencelibrary as build numbers 4 and 5 with build_url: https://releases.experiencenet.com/builds/soulmade/91/Builds/MercatorV0.9.zip.
  3. That URL returns HTTP/2 404 because hydrarelease only serves /api/v1/builds/<project>/<number> (see internal/api/server.go:111), not raw /builds/* paths. The HYDRARELEASE_MIRROR_URL proxy only covers hydrarelease's own project releases, not external build files.
  4. The mirror-a inventory has a placeholder entry for builds/soulmade/91/ with size: 0 and no actual file on disk at /var/lib/hydramirror/files/builds/soulmade/91/. GET /api/v1/builds/soulmade/91 on hydrarelease returns build soulmade/91 not found.
  5. Bodies in districts bxl1 and bxl1-test have been 404 looping since production_build was promoted earlier today. hydrabody.log lines: 
    [experiences] provisioning mercator-talks build #5
[experiences] downloading mercator-talks from https://releases.experiencenet.com/builds/soulmade/91/Builds/MercatorV0.9.zip...
[experiences] failed to install mercator-talks: downloading: HTTP 404

The previous successful build (CL 88 / V0.8, build #3) is registered with a mirror URL (https://mirror-a.experiencenet.com/api/v1/files/builds/soulmade/88/MercatorV0.8.zip). That URL also 404s today. So the mirror was always the serving path; the release server rsync destination has been dead since the mirror migration.

Workaround used today (manual, do not scale)

On the release server: curl -T /var/www/releases/builds/soulmade/91/Builds/MercatorV0.9.zip --upload-file -H "Authorization: Bearer <mirror-token>" https://mirror-a.experiencenet.com/api/v1/files/builds/soulmade/91/MercatorV0.9.zip, then POST /api/v1/builds/notify with the mirror URL, then promote dev to staging and staging to production. Bodies installed build 6 within 3 minutes.

Fix options

  1. Change hydraperforcewatcher to upload directly to hydramirror via PUT /api/v1/files/builds/<project>/<cl>/<file>.zip (stream via curl -T, no rsync), and emit build_url pointing at the mirror. This matches how hydrarelease publishes its own service binaries.
  2. Alternative: in hydrarelease, serve /var/www/releases/builds/* as static files, which just restores the old URLs. Less clean because the mirror is now the canonical storage.
  3. Alternative: add a linkMirrorFiles equivalent that runs when handleBuildNotify fires on hydraexperiencelibrary, reading from the release server and pushing to the mirror. Adds cross-service coupling.

Option 1 is the cleanest and matches existing patterns. The watcher already knows the mirror is the target (memory says "hydrarelease uploads binaries to mirror-a, NOT to local disk"); the rsync-to-/var/www/releases path is leftover from pre-mirror days.

Impact

Every new Perforce-driven experience build is silently broken at the fleet install layer until someone manually re-publishes the zip to the mirror. Experience library metadata looks healthy, which hides the problem. Today this broke mercator-talks V0.9 for three weeks (registered 2026-03-31, noticed only today 2026-04-21 when promoted to production).

Comments (2)

claude 21 Apr 2026 15:24

Implemented in 23235f2 and tagged v0.2.0. Pushed to origin main; GH Actions is building now (run 24730910436).

Next steps on the human side:

  1. Wait for CI to finish and confirm the watcher auto-updates on the visitflanders agent.
  2. Swap the agent config at ~/.hydraperforcewatcher/config.yaml: delete the release_server block, add mirror.{url,token}. Mirror token is in /etc/hydrarelease.env on 46.225.120.7 as HYDRARELEASE_MIRROR_TOKEN.
  3. Run docs/testbooks/mirror-upload.md against a test submission. On pass, move this issue to done.
claude 22 Apr 2026 16:18

Blocked on deploy step: agent-side config swap.

v0.2.0 binary is published and will auto-update on the visitflanders agent. However, v0.2.0 validates that mirror.url is present, so on the next restart it will fail to start unless the config is migrated first.

The perforce.visitflanders.experiencenet.com machine (88.198.195.179) is not in any of the three configured hcloud contexts (hydraexperiencenet, nimsforest, cederik) and the Claude environment SSH key is not authorized on it. Cannot complete the swap from here.

What is left:

  1. On perforce.visitflanders.experiencenet.com, edit ~/.hydraperforcewatcher/config.yaml:
    • Remove the release_server block.
    • Add mirror:\n url: https://mirror-a.experiencenet.com\n token: <HYDRARELEASE_MIRROR_TOKEN from /etc/hydrarelease.env on 46.225.120.7>.
  2. systemctl restart hydraperforcewatcher.
  3. Run docs/testbooks/mirror-upload.md against a scratch submission.
  4. Move this issue to done.

To unblock remote execution next time, either add a visitflanders hcloud context to ~/.config/hcloud/cli.toml (if the machine is on a separate Hetzner project) or authorize the Claude environment public key on the machine's root account.