HydraIssues

Summary

Introduce a new hydraskin role that turns any low-cost machine (Raspberry Pi 5, Intel N150 mini PC, etc.) into a container hosting node for the Hydra. Each container running on a hydraskin node is called a scale.

HydraSkin extends the Hydra with sub-500 USD CAPEX container hosts. The initial fleet is Raspberry Pi 5 (arm64) with NVMe storage, but the design is hardware-agnostic — any machine that can run Ubuntu Server + Incus can be a hydraskin node.

Motivation

We need lightweight, cheap compute nodes for running containerized workloads alongside the existing streaming/render infrastructure. Rather than building a custom container runtime, we leverage Incus (the community fork of LXD) which provides a mature REST API, CLI, and btrfs-backed storage — all managed through the existing HydraCluster recipe provisioning pipeline.

Design Principles

• Recipe-based provisioning — hydraskin is NOT a legacy service-binary role. It uses the existing recipe pipeline, so no changes needed in hydranode role switch, store.go, handlers_body.go, or heartbeat logic.
• Hardware-agnostic — the recipe supports both arm64 and amd64 architectures. Any Ubuntu Server machine can be a hydraskin node.
• No custom container API — Incus already provides a mature REST API on :8443 and a full CLI. Scales are managed directly through Incus tooling.
• Idempotent steps — every recipe step checks before acting, so reprovisioning is safe.

Changes Required

1. Register the role — pkg/api/handlers_api.go

Add to RoleCatalog: hydraskin with Category container.
Add container to CategoryOrder between creation and infra.

2. Create recipe — recipes/hydraskin-linux.yaml

Steps: check-prerequisites, install-incus, initialize-incus, configure-defaults, configure-firewall, setup-snapshot-cron, verify.

3. Add Skin view — hydrastreamingmonitor

Add Skin entry to More menu and /skin page showing hydraskin nodes and their scales (containers). Queries Incus REST API at wireguard_ip:8443 for each online skin node.

4. Move Pi provisioning script from iamneo to hydracluster

Rewrite for NVMe workflow with Ubuntu Server 24.04 arm64.

5. Separate issue: releases.experiencenet.com returns 404

releases.experiencenet.com returns 404 for hydranode/production/latest/hydranode-linux-arm64.

Provisioning Flow

Admin assigns hydraskin role to node -> Node heartbeat: ProvisionPending=true -> hydranode fetches GET /api/v1/body/provision -> Server loads recipes/hydraskin-linux.yaml -> hydranode executes steps -> Admin UI shows progress.

Affected Repositories

• cederikdotcom/hydracluster: Role catalog, recipe, Pi provision script
• cederikdotcom/hydrastreamingmonitor: Skin view (route, handler, templates)
• cederikdotcom/iamneo: Remove old Pi script, update runbook
• cederikdotcom/hydranode: Separate issue for releases.experiencenet.com 404

See full implementation plan at github.com/cederikdotcom/hydraskin