Description
## Goal
Deploy a district-local hydramirror fleet so bodies download experience builds from a mirror close to them, falling back to the central `mirror-a.experiencenet.com` on miss. Reduces cross-region bandwidth, cuts install time on multi-gigabyte Perforce builds, and lets venues keep streaming if the central mirror degrades.
## Current state
- `mirror-a.experiencenet.com` is the only running hydramirror instance. Every body hits it directly.
- The system already expects district mirrors: `hydramirror_url_pattern: "https://{district}.hydramirror.experiencenet.com"` is in `hydraexperiencelibrary` pipeline config.
- `bxl1.hydramirror.experiencenet.com` does not resolve (confirmed via pipeline health check: `lookup ... no such host`).
- `hydramirror` Go binary already exposes `GET /api/v1/peer/files/...` and `HEAD /api/v1/peer/files/...`, so peer-pull is designed in and just needs a configured origin.
- `hydrabody` currently uses `build_url` verbatim from the experience library. No host rewriting to a district-local mirror today.
## Depends on
- Issue #95 (hydraperforcewatcher publishes to mirror-a directly). That fix has to land first so we have a stable single origin before layering the fleet on top.
## Scope
1. **Deploy district mirror VMs**. One hydramirror per active district (`bxl1` first, additional districts as they come online). Start with a Hetzner cx23 in a location near the district. Provision via morpheus or existing infra pattern.
2. **Origin + peer config**. Each district mirror runs in peer mode with `origin_url: https://mirror-a.experiencenet.com` and its own admin token. Mirror-a remains the write-authoritative origin for uploads (only hydraperforcewatcher and hydrarelease PUT to it).
3. **DNS**. Add `{district}.hydramirror.experiencenet.com` A records for each deployed mirror. Hetzner DNS, same zone as the rest of experiencenet.com.
4. **TLS**. Autocert on each district mirror, same pattern as other services.
5. **hydrabody URL rewrite**. In the experience install code path, before calling `downloadFile(build_url, ...)`, rewrite the hostname to `{district}.hydramirror.experiencenet.com` if the body's configured district is non-empty and the URL host matches `mirror-a.experiencenet.com`. Fall back to the original URL on a non-200 HEAD or DNS failure.
6. **hydradistrict exposes district name to bodies**. If bodies do not already receive their district from `GET /api/v1/body/config`, add it.
7. **Health and observability**. hydraexperiencelibrary pipeline health check already queries `{district}.hydramirror.experiencenet.com/api/v1/health` per district. Once district mirrors are up, that check flips from `unreachable` to `healthy` automatically.
8. **Retention**. District mirrors are caches. Decide a retention policy (e.g., LRU over disk limit), and document it. Start simple with disk-size-bounded LRU.
## Out of scope
- Pre-warming builds on publish (push to all district mirrors on upload). Phase 2 optimisation if cold-miss install time on first-body-in-district is unacceptable.
- Geographic mirror selection beyond district. If a body's district has no mirror, it falls back to mirror-a, not to a nearer district's mirror.
- Write operations against district mirrors. They are read-through caches only.
## Acceptance criteria
- `bxl1.hydramirror.experiencenet.com` resolves and serves `GET /api/v1/health`.
- A body in district `bxl1` installing a fresh experience downloads the zip from `bxl1.hydramirror.experiencenet.com` (confirmed in hydrabody.log).
- Cold-cache install time is within the same order of magnitude as a direct `mirror-a` install. Warm-cache install is strictly faster.
- Pipeline health for `bxl1` shows `hydramirror: healthy` with the district URL.
- If `bxl1.hydramirror` goes offline, bodies fall back to `mirror-a` automatically. Streams keep working.
## Risks
- Adds a component in the hot install path. Misconfigured TLS, DNS, or peer-pull breaks installs. Mitigation: fallback to mirror-a on any error, verified in body code path and unit tests.
- Retention eviction may delete a build that a body is about to need. Mitigation: LRU plus large enough disk headroom, measure real hit rate.
- Divergence between district mirror contents and origin. Mitigation: peer-pull reads with SHA256 verification on each GET (hydramirror already tracks sha256).