HydraIssues

Goal

Define the district server clearly enough that a new district can be stood up from the
runbook, and so that scales running on hydraskin nodes have a public path in.

Six Hydra services are now published as multi-arch OCI images (hydranps, hydravenues,
hydraorganization, hydrapipeline, hydraexperiencelibrary, hydradistrict) and one runs as a
scale on pi-node-001, byte-for-byte identical to its Hetzner instance. None of them can
serve traffic, because nothing routes into a scale. That is the gap this closes.

The district server, as it actually exists

The model is one district server per ~100 km, serving nearby venues and bodies at low
latency. Brussels is the working example — OVH, EU-WEST-LZ-BRU-A:

Crucially, it already routes into the Pi network: a WireGuard peer carries
allowed-ips 10.0.0.0/24, and from the district box 10.0.0.4 and 10.0.0.5
(pi-node-003/004) are reachable over wg0. So the private path to the scales exists and
works today — only the public entry point is missing.

Proposal: the district server is also the ingress

hydrareverseproxy already does exactly this job — TLS termination with autocert and
domain-based routing — and is already in production on the dashboard server
(78.47.174.83, v0.1.2) fronting six domains. Its own config.example.yaml even routes
hydraorganization.experiencenet.com and hydravenues.experiencenet.com, so this pattern
was anticipated.

Target shape on a district machine:

:443  hydrareverseproxy (autocert, host-routed)
  ├─ hydraneckwebrtc-brussels.experiencenet.com -> 127.0.0.1:8443   (moved off :443)
  ├─ hydranps.experiencenet.com                 -> 10.0.0.4:8080    (scale, via wg0)
  └─ hydravenues.experiencenet.com              -> 10.0.0.5:8081    (scale, via wg0)

untouched: :3478 TURN | :49152-65535 relay | :47990 worker | :51820/udp wg

Each scale exposes its port on its hydraskin host with an Incus proxy device; the
district proxy routes to <pi-lan-ip>:<port> over the mesh.

Putting ingress on the district server is better than a central one: the box already
terminates the tunnel to its own Pis, so traffic reaches a scale in one local hop instead
of hairpinning through Hetzner. The public face and the private mesh endpoint are the same
machine, which is where ingress belongs.

Why the :443 conflict is safe to resolve

hydraneckwebrtc's controller currently owns :443 and would move behind the proxy.
The media path never touches the proxy — TURN (:3478), the relay range
(:49152-65535) and the worker (:47990) all bypass it. Only HTTP/WebSocket signalling is
proxied, and httputil.ReverseProxy handles WebSocket upgrades natively.

Header forwarding is no longer a concern: #63 (Authorization stripped) is fixed in
v0.1.2 — verified empirically, since the issue tracker's own authenticated API is served
through that proxy and returns 200 with a Bearer token. #63 can be closed.

Risk that needs a decision, not just noting

Brussels is a Local Zone with no Floating IP. Verified: /ip/failover is empty and the
region advertises instance and network but no floating-IP service. The precise
behaviour is the IP is fixed for the life of the instance but not portable — it does not
drift on its own, but a delete/recreate draws a new address from the pool. That is what
produced 141.227.136.12 -> 141.227.136.199 when the box was rebuilt as -v2.

The good news: almost nothing depends on that IP directly.

The district box is also the WireGuard hub for the district mesh — 24 peers (distinct
from the Hetzner hydraguard hub, which has 9 and matches the nodes carrying a
wireguard_ip in hydracluster). A hardcoded endpoint IP across 24 peers would make a
recreate genuinely painful.

It is not hardcoded. pkg/mesh/mesh.go:150 HubEndpoint() prefers endpoint_hostname
over endpoint, and the live mesh.yaml sets both:

hub:
  endpoint: 141.227.136.199
  endpoint_hostname: hydraguard.experiencenet.com

so peers are issued hydraguard.experiencenet.com:51820. Recreate recovery is therefore
one DNS update, not re-keying 24 peers.

Two cheap things that make a recreate routine

1. Script the DNS update. All records are in Hetzner DNS; hcloud zone rrset can
   update them in one command. Do this before adding scale routes — with ingress on this
   box, the blast radius grows from 4 records to every service in the district.
2. Add a WireGuard re-resolve timer. There is none today, and the kernel module caches
   the resolved endpoint rather than re-querying DNS. So after a DNS update peers only
   reconnect on their next tunnel restart. wireguard-tools ships reresolve-dns.sh; a
   small systemd timer makes recovery hands-off.

Backups of the mesh

Daily OVH snapshots exist — autobackup-brussels-district-v2, ~05:20 UTC, 7 days
retained. A restore recovers hub.key, mesh.yaml and wg0.conf together, so the mesh is
recoverable.

Three caveats:

• The on-box backups are not backups. /root/.hydraguard/backups/ holds timestamped
  mesh-*.yaml copies on the very instance they would protect.
• hub.key is the single irreplaceable secret (45 bytes, /etc/wireguard/hub.key).
  Lose it and all 24 peers must be re-keyed. It exists in exactly one place; nothing ships
  it off-box. The only other backup unit running is dpkg-db-backup, which is Debian's
  package database.
• Snapshots are same-zone (EU-WEST-LZ-BRU-A). That covers instance loss, corruption
  and bad config — not loss of the zone.
• One snapshot shows size: 0 (2026-07-25) where every other shows 5, suggesting a
  failed run that nothing alerted on.

A restore should be tested rather than assumed. An untested backup is a hypothesis,
and this one guards the credential that the whole district mesh depends on.

Documentation to correct

• hydracloudproviders/docs/runbooks/ovhcloud.md — documents the dead .12 instance, the
  old name and the old hydracluster node ID. Also worth recording that DNS for OVH
  instances lives in Hetzner DNS (it says this, and it is easy to miss).
• hydrareverseproxy/CLAUDE.md — claims it runs on "hydra-services cx23 (colocation
  server)", which does not exist in the Hetzner project. It actually runs on the dashboard
  server at 78.47.174.83.

Work

1. Correct the two runbooks above.
2. Deploy hydrareverseproxy on brussels-district-v2; move hydraneckwebrtc's controller
   to a local port and add it as the first route. Verify signalling and a real stream
   before adding anything else.
3. Add an Incus proxy device per scale on the hydraskin nodes.
4. Add scale routes; point DNS at the district server.
5. Write the district-server runbook so a second district is reproducible: what runs, what
   ports, what firewall rules, which DNS records, and the recreate procedure.

Related

• #406 hydraskin (the scales needing ingress)
• #408 Hydra migration (this unblocks the cutovers)
• #63 hydrareverseproxy Authorization — fixed in v0.1.2, close it