HydraIssues

## Goal

Define the **district server** clearly enough that a new district can be stood up from the
runbook, and so that scales running on hydraskin nodes have a public path in.

Six Hydra services are now published as multi-arch OCI images (hydranps, hydravenues,
hydraorganization, hydrapipeline, hydraexperiencelibrary, hydradistrict) and one runs as a
scale on pi-node-001, byte-for-byte identical to its Hetzner instance. **None of them can
serve traffic, because nothing routes into a scale.** That is the gap this closes.

## The district server, as it actually exists

The model is one district server per ~100 km, serving nearby venues and bodies at low
latency. Brussels is the working example — OVH, `EU-WEST-LZ-BRU-A`:

| | |
|---|---|
| Instance | `brussels-district-v2`, `141.227.136.199` |
| Spec | b3-8 · 2 vCPU · 16 GB · 92 GB free · x86_64 · Ubuntu 24.04 |
| hydracluster node | `node-2d5fba78` |
| Runs | hydraguard (wg0, **24 peers**), hydraneckwebrtc controller (:443) + worker (:47990), coturn (:3478), hydranode |

Crucially, it **already routes into the Pi network**: a WireGuard peer carries
`allowed-ips 10.0.0.0/24`, and from the district box `10.0.0.4` and `10.0.0.5`
(pi-node-003/004) are reachable over `wg0`. So the private path to the scales exists and
works today — only the public entry point is missing.

## Proposal: the district server is also the ingress

`hydrareverseproxy` already does exactly this job — TLS termination with autocert and
domain-based routing — and is **already in production** on the dashboard server
(78.47.174.83, v0.1.2) fronting six domains. Its own `config.example.yaml` even routes
`hydraorganization.experiencenet.com` and `hydravenues.experiencenet.com`, so this pattern
was anticipated.

Target shape on a district machine:

```
:443 hydrareverseproxy (autocert, host-routed)
├─ hydraneckwebrtc-brussels.experiencenet.com -> 127.0.0.1:8443 (moved off :443)
├─ hydranps.experiencenet.com -> 10.0.0.4:8080 (scale, via wg0)
└─ hydravenues.experiencenet.com -> 10.0.0.5:8081 (scale, via wg0)

untouched: :3478 TURN | :49152-65535 relay | :47990 worker | :51820/udp wg
```

Each scale exposes its port on its hydraskin host with an Incus `proxy` device; the
district proxy routes to `<pi-lan-ip>:<port>` over the mesh.

Putting ingress on the district server is better than a central one: the box already
terminates the tunnel to its own Pis, so traffic reaches a scale in one local hop instead
of hairpinning through Hetzner. The public face and the private mesh endpoint are the same
machine, which is where ingress belongs.

## Why the :443 conflict is safe to resolve

hydraneckwebrtc's controller currently owns :443 and would move behind the proxy.
**The media path never touches the proxy** — TURN (:3478), the relay range
(:49152-65535) and the worker (:47990) all bypass it. Only HTTP/WebSocket signalling is
proxied, and `httputil.ReverseProxy` handles WebSocket upgrades natively.

Header forwarding is no longer a concern: #63 (Authorization stripped) is **fixed** in
v0.1.2 — verified empirically, since the issue tracker's own authenticated API is served
through that proxy and returns 200 with a Bearer token. #63 can be closed.

## Risk that needs a decision, not just noting

**Brussels is a Local Zone with no Floating IP.** Verified: `/ip/failover` is empty and the
region advertises `instance` and `network` but no floating-IP service. The precise
behaviour is *the IP is fixed for the life of the instance but not portable* — it does not
drift on its own, but a delete/recreate draws a new address from the pool. That is what
produced `141.227.136.12` -> `141.227.136.199` when the box was rebuilt as `-v2`.

**The good news: almost nothing depends on that IP directly.**

The district box is also the **WireGuard hub for the district mesh — 24 peers** (distinct
from the Hetzner `hydraguard` hub, which has 9 and matches the nodes carrying a
`wireguard_ip` in hydracluster). A hardcoded endpoint IP across 24 peers would make a
recreate genuinely painful.

It is not hardcoded. `pkg/mesh/mesh.go:150` `HubEndpoint()` prefers `endpoint_hostname`
over `endpoint`, and the live `mesh.yaml` sets both:

```yaml
hub:
endpoint: 141.227.136.199
endpoint_hostname: hydraguard.experiencenet.com
```

so peers are issued `hydraguard.experiencenet.com:51820`. Recreate recovery is therefore
**one DNS update**, not re-keying 24 peers.

### Two cheap things that make a recreate routine

1. **Script the DNS update.** All records are in Hetzner DNS; `hcloud zone rrset` can
update them in one command. Do this *before* adding scale routes — with ingress on this
box, the blast radius grows from 4 records to every service in the district.
2. **Add a WireGuard re-resolve timer.** There is none today, and the kernel module caches
the resolved endpoint rather than re-querying DNS. So after a DNS update peers only
reconnect on their next tunnel restart. `wireguard-tools` ships `reresolve-dns.sh`; a
small systemd timer makes recovery hands-off.

## Backups of the mesh

**Daily OVH snapshots exist** — `autobackup-brussels-district-v2`, ~05:20 UTC, 7 days
retained. A restore recovers `hub.key`, `mesh.yaml` and `wg0.conf` together, so the mesh is
recoverable.

Three caveats:

- **The on-box backups are not backups.** `/root/.hydraguard/backups/` holds timestamped
`mesh-*.yaml` copies on the very instance they would protect.
- **`hub.key` is the single irreplaceable secret** (45 bytes, `/etc/wireguard/hub.key`).
Lose it and all 24 peers must be re-keyed. It exists in exactly one place; nothing ships
it off-box. The only other backup unit running is `dpkg-db-backup`, which is Debian's
package database.
- **Snapshots are same-zone** (`EU-WEST-LZ-BRU-A`). That covers instance loss, corruption
and bad config — not loss of the zone.
- **One snapshot shows `size: 0`** (2026-07-25) where every other shows 5, suggesting a
failed run that nothing alerted on.

**A restore should be tested rather than assumed.** An untested backup is a hypothesis,
and this one guards the credential that the whole district mesh depends on.

## Documentation to correct

- `hydracloudproviders/docs/runbooks/ovhcloud.md` — documents the dead `.12` instance, the
old name and the old hydracluster node ID. Also worth recording that DNS for OVH
instances lives in **Hetzner** DNS (it says this, and it is easy to miss).
- `hydrareverseproxy/CLAUDE.md` — claims it runs on "hydra-services cx23 (colocation
server)", which does not exist in the Hetzner project. It actually runs on the dashboard
server at 78.47.174.83.

## Work

1. Correct the two runbooks above.
2. Deploy `hydrareverseproxy` on `brussels-district-v2`; move hydraneckwebrtc's controller
to a local port and add it as the first route. Verify signalling and a real stream
before adding anything else.
3. Add an Incus `proxy` device per scale on the hydraskin nodes.
4. Add scale routes; point DNS at the district server.
5. Write the district-server runbook so a second district is reproducible: what runs, what
ports, what firewall rules, which DNS records, and the recreate procedure.

## Related

- #406 hydraskin (the scales needing ingress)
- #408 Hydra migration (this unblocks the cutovers)
- #63 hydrareverseproxy Authorization — fixed in v0.1.2, close it