#97 Deploy district-local hydramirror fleet with peer-pull from mirror-a

Goal

Deploy a district-local hydramirror fleet so bodies download experience builds from a mirror close to them, falling back to the central mirror-a.experiencenet.com on miss. Reduces cross-region bandwidth, cuts install time on multi-gigabyte Perforce builds, and lets venues keep streaming if the central mirror degrades.

Current state

mirror-a.experiencenet.com is the only running hydramirror instance. Every body hits it directly.
The system already expects district mirrors: hydramirror_url_pattern: "https://{district}.hydramirror.experiencenet.com" is in hydraexperiencelibrary pipeline config.
bxl1.hydramirror.experiencenet.com does not resolve (confirmed via pipeline health check: lookup ... no such host).
hydramirror Go binary already exposes GET /api/v1/peer/files/... and HEAD /api/v1/peer/files/..., so peer-pull is designed in and just needs a configured origin.
hydrabody currently uses build_url verbatim from the experience library. No host rewriting to a district-local mirror today.

Depends on

Issue #95 (hydraperforcewatcher publishes to mirror-a directly). That fix has to land first so we have a stable single origin before layering the fleet on top.

Scope

Deploy district mirror VMs. One hydramirror per active district (bxl1 first, additional districts as they come online). Start with a Hetzner cx23 in a location near the district. Provision via morpheus or existing infra pattern.
Origin + peer config. Each district mirror runs in peer mode with origin_url: https://mirror-a.experiencenet.com and its own admin token. Mirror-a remains the write-authoritative origin for uploads (only hydraperforcewatcher and hydrarelease PUT to it).
DNS. Add {district}.hydramirror.experiencenet.com A records for each deployed mirror. Hetzner DNS, same zone as the rest of experiencenet.com.
TLS. Autocert on each district mirror, same pattern as other services.
hydrabody URL rewrite. In the experience install code path, before calling downloadFile(build_url, ...), rewrite the hostname to {district}.hydramirror.experiencenet.com if the body's configured district is non-empty and the URL host matches mirror-a.experiencenet.com. Fall back to the original URL on a non-200 HEAD or DNS failure.
hydradistrict exposes district name to bodies. If bodies do not already receive their district from GET /api/v1/body/config, add it.
Health and observability. hydraexperiencelibrary pipeline health check already queries {district}.hydramirror.experiencenet.com/api/v1/health per district. Once district mirrors are up, that check flips from unreachable to healthy automatically.
Retention. District mirrors are caches. Decide a retention policy (e.g., LRU over disk limit), and document it. Start simple with disk-size-bounded LRU.

Out of scope

Pre-warming builds on publish (push to all district mirrors on upload). Phase 2 optimisation if cold-miss install time on first-body-in-district is unacceptable.
Geographic mirror selection beyond district. If a body's district has no mirror, it falls back to mirror-a, not to a nearer district's mirror.
Write operations against district mirrors. They are read-through caches only.

Acceptance criteria

bxl1.hydramirror.experiencenet.com resolves and serves GET /api/v1/health.
A body in district bxl1 installing a fresh experience downloads the zip from bxl1.hydramirror.experiencenet.com (confirmed in hydrabody.log).
Cold-cache install time is within the same order of magnitude as a direct mirror-a install. Warm-cache install is strictly faster.
Pipeline health for bxl1 shows hydramirror: healthy with the district URL.
If bxl1.hydramirror goes offline, bodies fall back to mirror-a automatically. Streams keep working.

Risks

Adds a component in the hot install path. Misconfigured TLS, DNS, or peer-pull breaks installs. Mitigation: fallback to mirror-a on any error, verified in body code path and unit tests.
Retention eviction may delete a build that a body is about to need. Mitigation: LRU plus large enough disk headroom, measure real hit rate.
Divergence between district mirror contents and origin. Mitigation: peer-pull reads with SHA256 verification on each GET (hydramirror already tracks sha256).

HydraIssues