HydraIssues

hydravenues has no backup: one Pi holds the only copy of the venue registry
open improvement Project: hydravenues Reporter: cederik 1 Sep 2026 09:57

Description

PROBLEM

The venue registry is the org-to-asset link for the whole platform. It is the single source that answers "which venues does this organization own", and the org dashboard (#597) cannot render anything without it. Its entire state is /srv/scales/hydravenues on pi-node-003-nvme, and there is no copy anywhere else.

This is not theoretical. On 2026-08-31 pi-node-003-nvme went offline (#602). hydravenues became unreachable, its edge route aged out of hydrascalerouter, and pending work that needed a venue write was blocked until the node came back. Had the disk failed rather than the node, the registry would be gone.

SCOPE

  1. Back up /srv/scales/hydravenues off the node on a schedule, with the same treatment for any other scale whose state exists in exactly one place (audit which those are; hydraissue at /srv/scales/hydraissue and hydraorganization at /srv/scales/hydraorganization are in the same position).
  2. Verify a restore actually works, on a test node, rather than assuming the archive is good.
  3. Document the restore in the hydravenues runbook: where the backup lives, how to put it back, and how to re-expose the scale and restore the edge route afterwards.

ACCEPTANCE

  • A backup exists off pi-node-003-nvme and is refreshed on a schedule.
  • A restore has been performed at least once on a test node and the service verified against /api/v1/health and a venue read.
  • The runbook carries the restore procedure.

NOTE

The scale rebuild path (hydraskin update) already survives because state lives on the host rather than in the container. That protects against a bad image, not against losing the host.