HydraIssues

## Summary

Assessment of migrating the 19-server Hydra stack off Hetzner Cloud to self-hosted infrastructure (Pi fleet + Incus workers). Current cost is approximately 85 EUR/month.

## Current Hetzner Inventory (Hydra Stack)

| Service | Specs | Monthly Cost | Notes |
|---------|-------|-------------|-------|
| hydracluster | 2c/4GB (CX23) | ~4.50 EUR | Control plane — **STAYS** (bootstrapping, see below) |
| hydrarelease | 2c/4GB (CX23) | ~4.50 EUR | Release server (releases.experiencenet.com) |
| hydraguard | 2c/4GB (CX23) | ~4.50 EUR | WireGuard mesh (managed by morpheus) |
| hydraexperiencelibrary | 2c/4GB (CX23) | ~4.50 EUR | |
| hydratransfer | 2c/4GB (CX23) | ~4.50 EUR | |
| hydrapipeline | 2c/4GB (CX23) | ~4.50 EUR | |
| hydraorganization | 2c/4GB (CX23) | ~4.50 EUR | |
| hydravenues | 2c/4GB (CX23) | ~4.50 EUR | |
| hydradistrict | 2c/4GB (CX23) | ~4.50 EUR | |
| hydraneck | 4c/8GB (CX33) | ~8.00 EUR | 8GB is for a co-located TP-Link Omada controller, not hydraneck (28MB) — see #413 |
| hydrahead | 2c/4GB (CX23) | ~4.50 EUR | |
| hydranps | 2c/4GB (CAX11 ARM) | ~3.60 EUR | Already running on ARM64 |
| hydrastreamingmonitor | 2c/4GB (CX23) | ~4.50 EUR | |
| hydraneckwebrtc | 2c/4GB (CX23) | ~4.50 EUR | WebRTC relay |
| hydraheadwebstream | 2c/4GB (CAX11 ARM) | ~3.60 EUR | Already running on ARM64 |
| hydraapplepipeline | 2c/4GB (CX23) | ~4.50 EUR | |
| hydraunrealengine-server | 2c/4GB (CX23) | ~4.50 EUR | |
| mirror-a | 2c/4GB (CX23) | ~4.50 EUR | |
| hydraperforce | 2c/4GB (CX23) | ~4.50 EUR | Perforce server |

## Migration Priority

### Phase 1 — Move first (lightweight management services)

These are pure Go binaries with minimal resource needs. No special network or storage requirements.

- hydrarelease
- hydraorganization
- hydravenues
- hydradistrict
- hydraexperiencelibrary
- hydratransfer
- hydrapipeline
- hydrabooks (if applicable)
- hydraneck — **moved up from Phase 2**, see #413: the service itself is only 28MB

### Phase 2 — Move with care

- hydrahead
- hydrastreamingmonitor
- hydranps (already on ARM, easy move)
- hydraheadwebstream (already on ARM, easy move)
- hydraguard (WireGuard mesh — moving this changes the network topology)

### Phase 3 — Keep on Hetzner (or move last)

These have special requirements that make self-hosting harder:

- **hydraneckwebrtc** — WebRTC needs low latency + high bandwidth at the edge. Moving to home/office network adds latency and NAT complexity
- **hydraperforce** — Storage-heavy, Perforce needs reliable disk I/O
- **hydraunrealengine-server** — May need specific resources
- **mirror-a** — Needs bandwidth for mirroring

## Technical Findings

- **No SQLite anywhere** — hydracluster uses pure YAML (nodes.yaml). Simple to move.
- **ARM64 proven** — hydranps and hydraheadwebstream already run on ARM (CAX11). Go cross-compiles cleanly.
- **ARM64 build gap** — some Makefiles only build linux/amd64. Need to add GOARCH=arm64 targets.
- **TLS consideration** — services bind port 80/443 with autocert. In LXC containers, better to run in --dev mode behind a host reverse proxy.
- **PTY dependency** — hydracluster uses /dev/ptmx for remote shell. LXC needs devpts mounted (most templates do this by default).
- **Auto-updater** — calls systemctl restart, so containers need systemd as init.
- **No blockers identified** for any Phase 1 service.

## Target Architecture

- **Pi fleet** (pi-node-001/002/003/004): hydraskin container hosts running Phase 1 services as scales. **Not** the hydracluster control plane — that stays on Hetzner (see the per-service plan).
- **Incus workers** (separate machines with more RAM): run containerized services as scales via hydraskin role
- **Storage**: Hetzner Storage Box at 3.20 EUR/month (NFS/SFTP) — 10.7 year break-even vs self-hosted NAS

## Cost Impact

Current Hetzner Hydra stack: ~85 EUR/month (~1020 EUR/year)
After migration: Pi hardware is one-time CAPEX (~400-600 EUR for the fleet), pays for itself in under a year. Remaining Hetzner cost only for services that must stay (WebRTC, Perforce).

---

## Update 2026-07-28 — hydraneck reclassified, see #413

hydraneck has been **moved from Phase 2 to Phase 1**, and the sizing question about its
host split into its own issue: **#413**.

Measured on the box: `hydraneck.service` uses **28 MB** (peak 31 MB). The cx33's 8 GB is
consumed by an unrelated TP-Link Omada WiFi controller co-tenanted on the same server
(Java `-Xmx1024m` at 1979 MB, plus MongoDB at 194 MB, in Docker). hydraneck is a
lightweight Go binary like the rest of Phase 1 and fits the hydraskin 512 MiB default
with roughly 18x headroom.

The Omada controller is a separate decision — third-party, stateful, and unrelated to
this migration except by co-location. Tracked in #413.


---

# Plan of action per service (2026-07-28)

Measured on the live boxes. Every Hydra service is a **single Go binary**, so
`GOOS=linux GOARCH=arm64` is a Makefile change away — the Pi fleet is genuinely
reachable for this stack.

## Target kind

Incus offers three instance kinds and they differ by ~2 orders of magnitude:
**OCI application container** (single process, ~1MB real overhead), **system
container** (boots systemd, ~15 processes, ~20MB), **VM** (own kernel, ~200MB+).
Measured on pi-node-001: `memory.current` 7.3MiB for an OCI container vs 142MiB
for an idle Ubuntu system container (anon 1.0MiB vs 11.6MiB).

A Go binary is one static process, so **OCI is the correct kind for nearly all of
this stack.**

## Per service

| Service | Measured RSS | Target | Kind | Notes |
|---|---|---|---|---|
| **hydracluster** | 22MB (cgroup 11MB) | **STAY** | — | Control plane. Circular dependency — see below. |
| hydrarelease | 20MB (peak 35MB) | Pi/worker | OCI | Also see #407 — its download path is currently broken |
| hydraneck | 28MB | Pi/worker | OCI | See #413 — the cx33 is sized for a co-located Omada controller, not this |
| hydrastreamingmonitor | 19MB (cgroup 9MB) | Pi/worker | OCI | |
| hydravenues | 19MB | Pi/worker | OCI | |
| hydraexperiencelibrary | 18MB | Pi/worker | OCI | |
| hydrapipeline | 17MB | Pi/worker | OCI | |
| hydraheadwebstream | 15MB | Pi/worker | OCI | already arm64 (cax11) — no rebuild needed |
| hydranps | not measured | Pi/worker | OCI | already arm64 (cax11) — no rebuild needed |
| hydraorganization | not measured | Pi/worker | OCI | expected ~20MB, same shape as siblings |
| hydratransfer | not measured | Pi/worker | OCI | file mover — give it disk/cache headroom |
| hydradistrict | not measured | Pi/worker | OCI | |
| hydrahead | not measured | Pi/worker | OCI | |
| hydraapplepipeline | not measured | assess | — | needs Apple/macOS runners, likely stays |
| **hydraguard** | svc 12MB, turnserver 32MB | **STAY** | — | Bootstrapping trap: recipes install WireGuard *through* the exec channel. If management depended on WG you could not repair WG. See #412. |
| **hydraneckwebrtc** | — | **STAY** | — | WebRTC needs low latency + bandwidth at the edge |
| **hydraperforce** | — | **STAY** | — | storage-heavy, needs reliable disk I/O |
| **hydraunrealengine-server** | — | **STAY** | — | resource requirements |
| **mirror-a** | hydramirror 40MB | **STAY (for now)** | — | needs bandwidth; also the redirect target that #407 is about |

## Blockers to clear first, in order

1. **#407 — releases.experiencenet.com serves nothing.** Every binary 404s. Both
the auto-updater and any recipe-based install depend on it. Nothing should move
until this is fixed.
2. **arm64 builds.** Add `GOARCH=arm64` targets to the Makefiles that lack them.
Cheap for Go, but must be done before anything lands on a Pi.
3. **The update model changes with OCI.** This issue previously noted "Auto-updater
calls systemctl restart, so containers need systemd as init" — that assumed
system containers. OCI containers have no init; you update by replacing the
image. Decide this **before** Phase 1, not during it.
4. **#412 — 17 of 29 online nodes run hydranode v1.10.21 or older**, which makes
remote management ~25s per command instead of ~2s. Worth clearing first simply
to make the migration itself less painful.

## Capacity note

pi-node-001 has 7.7GB; pi-node-003/004 have **3.9GB**. At ~45MB real per OCI scale
(≈20MB container + ≈25MB service) the whole Phase 1 set fits one Pi several times
over — memory is not the constraint. Caps are not reservations, so the 512MiB
default can be overcommitted freely.


---

## hydracluster stays on Hetzner (decision, 2026-07-28)

Previously listed as "move last". Reclassified to **STAY** — the dependency is
circular, not merely awkward about ordering.

hydracluster is what provisions and manages hydraskin nodes. If it ran as a scale
on one of them:

- **It provisions its own host.** Bringing up or repairing the node it lives on
requires the thing living on that node.
- **It is the exec channel.** `POST /api/v1/nodes/{id}/exec` is how every node in
the fleet is managed remotely. Losing hydracluster means losing remote management
of the entire fleet, including the ability to diagnose why it is down.
- **Recovery needs physical access.** The Pis sit on private LANs with no public
address and no inbound SSH (pi-node-001 is 192.168.68.58). If the control plane
is unreachable there is no second path in — recovery becomes a site visit.
- **It holds the fleet state.** `nodes.yaml` is the source of truth for enrollment,
roles and tokens.

This is the same trap already flagged for **hydraguard**, whose WireGuard config is
installed *through* the exec channel — you would need the tunnel up to repair the
tunnel. hydracluster is the more severe case because it is the layer hydraguard's
repair path itself depends on.

Keeping the control plane on infrastructure it does not manage is the point: it is
the fixed reference the self-hosted fleet is managed *from*. The ~4.50 EUR/month is
cheap for a recovery path that does not require driving somewhere.

**Revised savings:** hydracluster joins hydraguard, hydraneckwebrtc, hydraperforce,
hydraunrealengine-server and mirror-a as the Hetzner-resident set.