Description
## Overview
Migrate the Hydra and NimsForest management stacks from Hetzner Cloud to a self-hosted Raspberry Pi 5 fleet running Incus/LXC containers.
## Current Hetzner Infrastructure
### Hydra stack (19 servers, mostly CX23 2c/4GB)
hydracluster, hydrarelease, hydraguard, hydraexperiencelibrary, hydratransfer, hydrapipeline, hydraorganization, hydravenues, hydradistrict, hydraneck (4c/8GB), hydrahead, hydranps (ARM), hydrastreamingmonitor, hydraneckwebrtc, hydraheadwebstream (ARM), hydraapplepipeline, hydraunrealengine-server, mirror-a, hydraperforce
### NimsForest stack (10 servers, mostly CX23 2c/4GB)
land-nimsforest-one, docker-registry, neoremote (4c/8GB), land-shared-one, land-executxr-one, land-experiencenet-one, land-xrvalley-one, land-thenaturalbeautyclub-one, land-riseof-one, land-callyouragentai-one
## Target Pi Fleet
| Node | RAM | NVMe | Location |
|------|-----|------|----------|
| pi-node-001 | 8GB | 1x 256GB | Primary / entry point |
| pi-node-002 | 4GB | 2x 256GB | Redundant worker, location A |
| pi-node-003 | 4GB | 2x 256GB | Redundant worker, location B |
| Total | 16GB | ~1TB | Geo-distributed |
Each node runs Ubuntu Server 24.04 + Incus with NVMe-backed storage pools.
## Architecture
- Hydra WireGuard mesh for networking between nodes
- NATS JetStream clustering for NimsForest state replication
- Litestream for any SQLite replication needs
- Services run as LXC containers on Incus
- Auto-enrollment via cloud-init + hydranode
## Findings from Code Analysis
### HydraCluster
- Pure Go, no CGo, no SQLite (YAML store only)
- TLS via built-in autocert (ports 80/443)
- Uses PTY for remote shell (needs /dev/ptmx in LXC)
- Auto-updater via hydrarelease
### NimsForest (nimsforest2)
- Pure Go (CGO_ENABLED=0), embedded NATS server with JetStream
- Storage: NATS KV (Soil) + Streams (River, Humus, Taproot)
- TLS handled by separate nimsforestnginx binary
- Clustering via IPv6 NATS routes
## Required Work
### Must do before migration
1. **ARM64 builds** - Both repos only build linux/amd64. Add GOARCH=arm64 targets to Makefiles and CI/release pipelines. Two services (hydranps, hydraheadwebstream) already run on ARM (CAX11).
2. **Auto-updater architecture awareness** - Ensure hydrarelease publishes ARM64 binaries and the updater downloads the correct arch.
3. **IPv6 in LXC** - NimsForest requires IPv6 (fatals without it). Configure IPv6 on Incus network bridge.
4. **TLS strategy** - Run services in --dev/--listen mode behind host-level reverse proxy, or use privileged containers with CAP_NET_BIND_SERVICE.
### May need adjustment
5. **systemd hardening** - NimsForest uses MemoryDenyWriteExecute=true which may fail in some LXC configs.
6. **/mnt/forest mount** - NimsForest expects this for registry/file storage. Needs bind mount from host NVMe.
7. **File descriptor limits** - Both services set LimitNOFILE=65536. Verify LXC allows this.
### Services to keep on Hetzner
- hydraneckwebrtc, hydraheadwebstream (WebRTC needs low latency + bandwidth)
- hydraperforce (storage-heavy)
- neoremote (8GB dev box)
- hydraunrealengine-server (GPU workloads)
## Migration Order (suggested)
1. hydracluster, hydrarelease, hydraorganization, hydravenues, hydradistrict (lightweight mgmt)
2. land-nimsforest-one, land-shared-one (core NimsForest)
3. Remaining hydra services
4. Per-tenant lands