Overview
Migrate the Hydra and NimsForest management stacks from Hetzner Cloud to a self-hosted Raspberry Pi 5 fleet running Incus/LXC containers.
Current Hetzner Infrastructure
Hydra stack (19 servers, mostly CX23 2c/4GB)
hydracluster, hydrarelease, hydraguard, hydraexperiencelibrary, hydratransfer, hydrapipeline, hydraorganization, hydravenues, hydradistrict, hydraneck (4c/8GB), hydrahead, hydranps (ARM), hydrastreamingmonitor, hydraneckwebrtc, hydraheadwebstream (ARM), hydraapplepipeline, hydraunrealengine-server, mirror-a, hydraperforce
NimsForest stack (10 servers, mostly CX23 2c/4GB)
land-nimsforest-one, docker-registry, neoremote (4c/8GB), land-shared-one, land-executxr-one, land-experiencenet-one, land-xrvalley-one, land-thenaturalbeautyclub-one, land-riseof-one, land-callyouragentai-one
Target Pi Fleet
| Node |
RAM |
NVMe |
Location |
| pi-node-001 |
8GB |
1x 256GB |
Primary / entry point |
| pi-node-002 |
4GB |
2x 256GB |
Redundant worker, location A |
| pi-node-003 |
4GB |
2x 256GB |
Redundant worker, location B |
| Total |
16GB |
~1TB |
Geo-distributed |
Each node runs Ubuntu Server 24.04 + Incus with NVMe-backed storage pools.
Architecture
- Hydra WireGuard mesh for networking between nodes
- NATS JetStream clustering for NimsForest state replication
- Litestream for any SQLite replication needs
- Services run as LXC containers on Incus
- Auto-enrollment via cloud-init + hydranode
Findings from Code Analysis
HydraCluster
- Pure Go, no CGo, no SQLite (YAML store only)
- TLS via built-in autocert (ports 80/443)
- Uses PTY for remote shell (needs /dev/ptmx in LXC)
- Auto-updater via hydrarelease
NimsForest (nimsforest2)
- Pure Go (CGO_ENABLED=0), embedded NATS server with JetStream
- Storage: NATS KV (Soil) + Streams (River, Humus, Taproot)
- TLS handled by separate nimsforestnginx binary
- Clustering via IPv6 NATS routes
Required Work
Must do before migration
- ARM64 builds - Both repos only build linux/amd64. Add GOARCH=arm64 targets to Makefiles and CI/release pipelines. Two services (hydranps, hydraheadwebstream) already run on ARM (CAX11).
- Auto-updater architecture awareness - Ensure hydrarelease publishes ARM64 binaries and the updater downloads the correct arch.
- IPv6 in LXC - NimsForest requires IPv6 (fatals without it). Configure IPv6 on Incus network bridge.
- TLS strategy - Run services in --dev/--listen mode behind host-level reverse proxy, or use privileged containers with CAP_NET_BIND_SERVICE.
May need adjustment
- systemd hardening - NimsForest uses MemoryDenyWriteExecute=true which may fail in some LXC configs.
- /mnt/forest mount - NimsForest expects this for registry/file storage. Needs bind mount from host NVMe.
- File descriptor limits - Both services set LimitNOFILE=65536. Verify LXC allows this.
Services to keep on Hetzner
- hydraneckwebrtc, hydraheadwebstream (WebRTC needs low latency + bandwidth)
- hydraperforce (storage-heavy)
- neoremote (8GB dev box)
- hydraunrealengine-server (GPU workloads)
Migration Order (suggested)
- hydracluster, hydrarelease, hydraorganization, hydravenues, hydradistrict (lightweight mgmt)
- land-nimsforest-one, land-shared-one (core NimsForest)
- Remaining hydra services
- Per-tenant lands