Summary

Assessment of migrating the 19-server Hydra stack off Hetzner Cloud to self-hosted infrastructure (Pi fleet + Incus workers). Current cost is approximately 85 EUR/month.

Current Hetzner Inventory (Hydra Stack)

Migration Priority

Phase 1 — Move first (lightweight management services)

These are pure Go binaries with minimal resource needs. No special network or storage requirements.

• hydrarelease
• hydraorganization
• hydravenues
• hydradistrict
• hydraexperiencelibrary
• hydratransfer
• hydrapipeline
• hydrabooks (if applicable)
• hydraneck — moved up from Phase 2, see #413: the service itself is only 28MB

Phase 2 — Move with care

• hydrahead
• hydrastreamingmonitor
• hydranps (already on ARM, easy move)
• hydraheadwebstream (already on ARM, easy move)
• hydraguard (WireGuard mesh — moving this changes the network topology)

Phase 3 — Keep on Hetzner (or move last)

These have special requirements that make self-hosting harder:

• hydraneckwebrtc — WebRTC needs low latency + high bandwidth at the edge. Moving to home/office network adds latency and NAT complexity
• hydraperforce — Storage-heavy, Perforce needs reliable disk I/O
• hydraunrealengine-server — May need specific resources
• mirror-a — Needs bandwidth for mirroring

Technical Findings

• No SQLite anywhere — hydracluster uses pure YAML (nodes.yaml). Simple to move.
• ARM64 proven — hydranps and hydraheadwebstream already run on ARM (CAX11). Go cross-compiles cleanly.
• ARM64 build gap — some Makefiles only build linux/amd64. Need to add GOARCH=arm64 targets.
• TLS consideration — services bind port 80/443 with autocert. In LXC containers, better to run in --dev mode behind a host reverse proxy.
• PTY dependency — hydracluster uses /dev/ptmx for remote shell. LXC needs devpts mounted (most templates do this by default).
• Auto-updater — calls systemctl restart, so containers need systemd as init.
• No blockers identified for any Phase 1 service.

Target Architecture

• Pi fleet (pi-node-001/002/003/004): hydraskin container hosts running Phase 1 services as scales. Not the hydracluster control plane — that stays on Hetzner (see the per-service plan).
• Incus workers (separate machines with more RAM): run containerized services as scales via hydraskin role
• Storage: Hetzner Storage Box at 3.20 EUR/month (NFS/SFTP) — 10.7 year break-even vs self-hosted NAS

Cost Impact

Current Hetzner Hydra stack: ~85 EUR/month (~1020 EUR/year)
After migration: Pi hardware is one-time CAPEX (~400-600 EUR for the fleet), pays for itself in under a year. Remaining Hetzner cost only for services that must stay (WebRTC, Perforce).

Update 2026-07-28 — hydraneck reclassified, see #413

hydraneck has been moved from Phase 2 to Phase 1, and the sizing question about its
host split into its own issue: #413.

Measured on the box: hydraneck.service uses 28 MB (peak 31 MB). The cx33's 8 GB is
consumed by an unrelated TP-Link Omada WiFi controller co-tenanted on the same server
(Java -Xmx1024m at 1979 MB, plus MongoDB at 194 MB, in Docker). hydraneck is a
lightweight Go binary like the rest of Phase 1 and fits the hydraskin 512 MiB default
with roughly 18x headroom.

The Omada controller is a separate decision — third-party, stateful, and unrelated to
this migration except by co-location. Tracked in #413.

Plan of action per service (2026-07-28)

Measured on the live boxes. Every Hydra service is a single Go binary, so
GOOS=linux GOARCH=arm64 is a Makefile change away — the Pi fleet is genuinely
reachable for this stack.

Target kind

Incus offers three instance kinds and they differ by ~2 orders of magnitude:
OCI application container (single process, ~1MB real overhead), system
container (boots systemd, ~15 processes, ~20MB), VM (own kernel, ~200MB+).
Measured on pi-node-001: memory.current 7.3MiB for an OCI container vs 142MiB
for an idle Ubuntu system container (anon 1.0MiB vs 11.6MiB).

A Go binary is one static process, so OCI is the correct kind for nearly all of
this stack.

Per service

Blockers to clear first, in order

1. #407 — releases.experiencenet.com serves nothing. Every binary 404s. Both
   the auto-updater and any recipe-based install depend on it. Nothing should move
   until this is fixed.
2. arm64 builds. Add GOARCH=arm64 targets to the Makefiles that lack them.
   Cheap for Go, but must be done before anything lands on a Pi.
3. The update model changes with OCI. This issue previously noted "Auto-updater
   calls systemctl restart, so containers need systemd as init" — that assumed
   system containers. OCI containers have no init; you update by replacing the
   image. Decide this before Phase 1, not during it.
4. #412 — 17 of 29 online nodes run hydranode v1.10.21 or older, which makes
   remote management ~25s per command instead of ~2s. Worth clearing first simply
   to make the migration itself less painful.

Capacity note

pi-node-001 has 7.7GB; pi-node-003/004 have 3.9GB. At ~45MB real per OCI scale
(≈20MB container + ≈25MB service) the whole Phase 1 set fits one Pi several times
over — memory is not the constraint. Caps are not reservations, so the 512MiB
default can be overcommitted freely.

hydracluster stays on Hetzner (decision, 2026-07-28)

Previously listed as "move last". Reclassified to STAY — the dependency is
circular, not merely awkward about ordering.

hydracluster is what provisions and manages hydraskin nodes. If it ran as a scale
on one of them:

• It provisions its own host. Bringing up or repairing the node it lives on
  requires the thing living on that node.
• It is the exec channel. POST /api/v1/nodes/{id}/exec is how every node in
  the fleet is managed remotely. Losing hydracluster means losing remote management
  of the entire fleet, including the ability to diagnose why it is down.
• Recovery needs physical access. The Pis sit on private LANs with no public
  address and no inbound SSH (pi-node-001 is 192.168.68.58). If the control plane
  is unreachable there is no second path in — recovery becomes a site visit.
• It holds the fleet state. nodes.yaml is the source of truth for enrollment,
  roles and tokens.

This is the same trap already flagged for hydraguard, whose WireGuard config is
installed through the exec channel — you would need the tunnel up to repair the
tunnel. hydracluster is the more severe case because it is the layer hydraguard's
repair path itself depends on.

Keeping the control plane on infrastructure it does not manage is the point: it is
the fixed reference the self-hosted fleet is managed from. The ~4.50 EUR/month is
cheap for a recovery path that does not require driving somewhere.

Revised savings: hydracluster joins hydraguard, hydraneckwebrtc, hydraperforce,
hydraunrealengine-server and mirror-a as the Hetzner-resident set.