HydraIssues

Deploy hydraperforcewatcher v0.2.0 to visitflanders and unstick the auto-updater
open bug Project: hydraperforcewatcher Reporter: cederik 23 Apr 2026 13:30

Description

## Goal

Get hydraperforcewatcher v0.2.0 running on the visitflanders Perforce agent, so newly-submitted Perforce changelists publish to hydramirror (not the dead /var/www/releases path) and bodies can actually download them. The v0.2.0 binary is already published on releases.experiencenet.com (see #95); this issue is the deploy-side work that #95 did not cover.

## Observed state (as of 2026-04-23)

- `releases.experiencenet.com/hydraperforcewatcher/production/latest.json` -> `{"version":"0.2.0"}` (CI run 24730910436 succeeded 2026-04-21).
- visitflanders agent is still running `agent_version: v0.1.3` per hydraperforce state.yaml. That version predates even v0.1.4, which shipped 2026-03-24. So the auto-updater has been inactive on this agent for at least a month.
- Consequence: CL 92 "Add build v0.95" was processed on 2026-04-22 by the old code path. MercatorV0.95.zip sits at /var/www/releases/builds/soulmade/92/Builds/MercatorV0.95.zip on the release server but was never uploaded to hydramirror. mercator-talks has `development_build: 7` in hydraexperiencelibrary with a broken `build_url: https://releases.experiencenet.com/builds/soulmade/92/`. If promoted to production today, bodies will 404 the same way they did for CL 91.
- I cannot reach the visitflanders machine from the Claude environment. It is not in any of the three configured hcloud contexts (hydraexperiencenet, nimsforest, cederik) and the environment SSH key is not authorized on the machine.

## Scope

1. **Diagnose why the auto-updater is stuck** on the visitflanders agent. Candidate causes:
- Service not restarting after an update (updater swaps the binary on disk but depends on a service restart to pick it up).
- Stale version pin in a config field.
- Network/TLS issue when the agent reaches releases.experiencenet.com (would show in journalctl).
- WDAC or code-signing rejecting the new binary on a Windows host (we think this is a Linux box but confirm).
2. **Migrate the agent config** at `~/.hydraperforcewatcher/config.yaml` to the v0.2.0 schema. Delete the `release_server:` block, add:
```yaml
mirror:
url: "https://mirror-a.experiencenet.com"
token: "<HYDRARELEASE_MIRROR_TOKEN from /etc/hydrarelease.env on 46.225.120.7>"
```
3. **Restart the service** and confirm it starts clean: `systemctl status hydraperforcewatcher && journalctl -u hydraperforcewatcher -n 50`.
4. **Confirm v0.2.0 is running**: next `POST /api/v1/agents/visitflanders/state` should report `agent_version: v0.2.0` (visible in hydraperforce state.yaml on 138.199.195.170).
5. **Patch CL 92 manually** using the same procedure as the mercator-talks V0.9 fix on 2026-04-21:
- On releases server: `curl -T /var/www/releases/builds/soulmade/92/Builds/MercatorV0.95.zip -H "Authorization: Bearer $HYDRARELEASE_MIRROR_TOKEN" --upload-file https://mirror-a.experiencenet.com/api/v1/files/builds/soulmade/92/MercatorV0.95.zip`
- POST /api/v1/builds/notify on hydraexperiencelibrary with the mirror URL, so a new build entry (build #8) is created with a valid URL.
- Update exe_path on the experience if the zip folder name differs from `MercatorV0.9` (likely now `MercatorV0.95` or similar).
- Promote dev to staging to production when the developer is ready.
6. **Run docs/testbooks/mirror-upload.md** end-to-end on a scratch submission to confirm the v0.2.0 path works.

## Unblocking remote access (one-time, for future work)

Either:
- Add a `visitflanders` hcloud context to `~/.config/hcloud/cli.toml` on the Claude box if the machine is on a separate Hetzner project. Then `HCLOUD_CONTEXT=visitflanders hcloud server ssh hydraperforcewatcher` (or whatever the server is named) would work.
- Add `/home/claude-user/.ssh/id_ed25519.pub` to `/root/.ssh/authorized_keys` on the visitflanders machine. Then plain `ssh root@perforce.visitflanders.experiencenet.com` works.

Without one of these, this class of deploy work keeps requiring a human hand.

## Acceptance

- `hydraperforce` state.yaml shows `agent_version: v0.2.0` for `visitflanders`.
- A fresh test changelist flows end-to-end per docs/testbooks/mirror-upload.md.
- mercator-talks is live on V0.95 (build #8 via the manual patch from step 5) with a mirror URL and bodies can install it.

## Related

- #95: code fix that this issue deploys.
- #97: district-local hydramirror fleet (separate scope).