HydraIssues

Issues / #602
hydravenues down: pi-node-003-nvme offline, no edge route
done bug Project: hydravenues Reporter: cederik 29 Aug 2026 21:54

Description

FOUND 2026-08-29 while reassigning venue ad6 to the new hydra organization.

SYMPTOMS

  • https://hydravenues.experiencenet.com returns 404 from the edge (141.227.136.199) on every path, including /api/v1/health.
  • The hydrascalerouter routes.json has NO route for hydravenues; the route aged out.
  • hydracluster shows pi-node-003-nvme (node-6bf57aed, LAN 192.168.68.52) OFFLINE. pi-node-002-c is also offline.
  • The hydravenues scale and its state (/srv/scales/hydravenues) live on that node, so the venue registry and its data are unreachable. No exec channel while the node is offline.

IMPACT

  • Venue reads used by dashboards and ops (GET /api/v1/venues?organization=...) fail.
  • BLOCKED WORK: set organization_id=hydra on venue ad6 (the hydra org exists in hydraorganization since 2026-08-29). Do this via the hydravenues API after recovery, never by editing YAML on the server.

RECOVERY

  1. Physical check of pi-node-003-nvme at its location (LAN 192.168.68.52): power, network, SD/NVMe.
  2. When the node is back: confirm hydranode reports online, the hydravenues scale is RUNNING, and the edge route reappears in routes.json.
  3. Then PATCH venue ad6 to organization hydra and verify GET /api/v1/venues?organization=visit-flanders no longer returns ad6.
  4. Consider: the venue registry has no redundancy; a single Pi holds the only copy of the venue data. Evaluate a backup job for /srv/scales/hydravenues.

Comments (1)

claude 31 Aug 2026 20:17

RESOLVED 2026-08-31. pi-node-003-nvme is back online and hydravenues is reachable again (edge route restored, GET /api/v1/health 200). The blocked work is done: venue ad6 now has organization_id=hydra (PUT /api/v1/venues/ad6 with the admin token read on the node, so the token never left it), and its stale description was corrected from "Visit Flanders venue in Overijse" to "Hydra internal venue in Overijse". Verified: GET /api/v1/venues?organization=visit-flanders now returns cloud-seven, rupelmonde, mobile-kit, sint-niklaas-tourism-office; GET /api/v1/venues?organization=hydra returns ad6. The hydra organization was created in hydraorganization on 2026-08-29 (type internal). Still open from this issue: the venue registry has no redundancy, one Pi holds the only copy of /srv/scales/hydravenues. Worth a backup job.