HydraIssues

## Summary

Introduce a new **hydraskin** role that turns any low-cost machine (Raspberry Pi 5, Intel N150 mini PC, etc.) into a container hosting node for the Hydra. Each container running on a hydraskin node is called a **scale**.

HydraSkin extends the Hydra with sub-500 USD CAPEX container hosts. The initial fleet is Raspberry Pi 5 (arm64) with NVMe storage, but the design is hardware-agnostic — any machine that can run Ubuntu Server + Incus can be a hydraskin node.

## Motivation

We need lightweight, cheap compute nodes for running containerized workloads alongside the existing streaming/render infrastructure. Rather than building a custom container runtime, we leverage **Incus** (the community fork of LXD) which provides a mature REST API, CLI, and btrfs-backed storage — all managed through the existing HydraCluster recipe provisioning pipeline.

## Design Principles

- **Recipe-based provisioning** — hydraskin is NOT a legacy service-binary role. It uses the existing recipe pipeline, so no changes needed in hydranode role switch, store.go, handlers_body.go, or heartbeat logic.
- **Hardware-agnostic** — the recipe supports both arm64 and amd64 architectures. Any Ubuntu Server machine can be a hydraskin node.
- **No custom container API** — Incus already provides a mature REST API on :8443 and a full CLI. Scales are managed directly through Incus tooling.
- **Idempotent steps** — every recipe step checks before acting, so reprovisioning is safe.

## Changes Required

### 1. Register the role — pkg/api/handlers_api.go

Add to RoleCatalog: hydraskin with Category container.
Add container to CategoryOrder between creation and infra.

### 2. Create recipe — recipes/hydraskin-linux.yaml

Steps: check-prerequisites, install-incus, initialize-incus, configure-defaults, configure-firewall, setup-snapshot-cron, verify.

### 3. Add Skin view — hydrastreamingmonitor

Add Skin entry to More menu and /skin page showing hydraskin nodes and their scales (containers). Queries Incus REST API at wireguard_ip:8443 for each online skin node.

### 4. Move Pi provisioning script from iamneo to hydracluster

Rewrite for NVMe workflow with Ubuntu Server 24.04 arm64.

### 5. Separate issue: releases.experiencenet.com returns 404

releases.experiencenet.com returns 404 for hydranode/production/latest/hydranode-linux-arm64.

## Provisioning Flow

Admin assigns hydraskin role to node -> Node heartbeat: ProvisionPending=true -> hydranode fetches GET /api/v1/body/provision -> Server loads recipes/hydraskin-linux.yaml -> hydranode executes steps -> Admin UI shows progress.

## Affected Repositories

- cederikdotcom/hydracluster: Role catalog, recipe, Pi provision script
- cederikdotcom/hydrastreamingmonitor: Skin view (route, handler, templates)
- cederikdotcom/iamneo: Remove old Pi script, update runbook
- cederikdotcom/hydranode: Separate issue for releases.experiencenet.com 404

See full implementation plan at github.com/cederikdotcom/hydraskin