HydraIssues

Context

The use case is that people host microservices on their Hydra, some of which need to be publicly reachable. The mechanics of the path are proven — district hub -> venue tunnel -> Pi LAN address -> Incus proxy device -> scale, measured at HTTP 200 in 73 ms. Automating ingress is tracked separately.

This issue is for the questions that automation does not answer, and which should be decided before anything customer-facing is switched on.

1. Whose domains, and who may claim one?

If tenants bring their own domain they CNAME to the district server and autocert issues a certificate on first request. Two consequences:

• Anyone who points DNS at us can trigger a certificate request. A dynamic HostPolicy must check registration, not merely accept whatever arrives, or the box becomes an open ACME client and will hit Let's Encrypt rate limits.
• Nothing currently stops one tenant claiming another's domain. Incus projects already isolate the containers; the domain namespace has no equivalent owner check. First-come-first-served across tenants is not acceptable once there is more than one.

Decide: are domains only ever *.experiencenet.com subdomains we control, or do we accept customer domains? The second needs a verification step (DNS TXT or similar) before a route is honoured.

2. Public traffic transits the district hub

The Pi is LAN-private behind NAT, so the mesh is the only inbound route and the hub is always on the path. That means:

• Customer traffic consumes venue uplink and district hub bandwidth. A popular service on a Pi at a venue is paid for twice.
• It places brussels-district-v2 on the public path for customer workloads. That is the box whose backup is still unproven (#422), whose snapshots fail roughly 1 in 5 silently (#424), and which also runs hydraneckwebrtc and the WireGuard hub for 24 peers.
• It is a DDoS surface: public traffic to a customer service lands on the district box, and the mesh depends on that box staying up.

Decide whether public workloads should terminate somewhere other than the district hub, or whether the hub is hardened and monitored first.

3. One backend per domain, no health check, no failover

hydrareverseproxy builds one httputil.NewSingleHostReverseProxy per domain. There is no pool, no health check and no failover: a scale that dies is a 502 until someone notices.

That is tolerable for internal tools and weak for anything customer-facing. It also interacts with the state problem — most Hydra services are single-writer YAML stores, so "just run two" would split writes and diverge. Capacity and availability need state solved first, or need services that are genuinely stateless.

4. Resource isolation is capped but not guaranteed

Scales inherit 512 MiB memory and 5 GiB disk caps, kernel-enforced (verified: a 700 MB allocation in a 512 MiB scale is OOM-killed inside its own cgroup, node unaffected). But CPU is deliberately unbounded — limits.cpu in Incus is cpuset pinning rather than a share, and pinning would cripple Go services by forcing GOMAXPROCS=1.

So one tenant's busy scale can starve a neighbour's. Acceptable between internal services on a node we control; a different proposition when tenants are paying.

Also note disk caps only work because btrfs quota enable is run at provision time — without it a size: is accepted silently and enforces nothing.

5. Registry access for tenant images

If tenants deploy their own images they need to push somewhere. Today scaleregistry.experiencenet.com has a single shared credential distributed to nodes, with no per-tenant namespacing and no per-repository authorisation. Related maturity work is #427.

6. Blast radius of the alternative

For completeness, the rejected option: making scales full WireGuard mesh members would give every scale reachability from all 24 peers — iPads, Windows air units, venue routers. Public hosting does not need internal reachability, and this would widen exposure considerably for no benefit to this use case. Recorded so the decision is not revisited by accident.

Suggested position

Start with *.experiencenet.com subdomains only, allocated by us, on services we operate. That removes questions 1 and 5 entirely and makes the ingress automation useful immediately. Revisit customer domains and tenant image pushes only once ownership and verification exist.