PROBLEM
Two shared tokens were rotated in the last week and neither rotation had a way to reach everything that holds a copy. The holders only find out when something fails.
- scaleregistry push/pull token, rotated 2026-08-28 around 07:29 UTC. It landed in the middle of a hydraissue image push and failed it. Every copy dated before that rotation was dead until refreshed by hand.
- hydracluster admin token, rotated 2026-08-31 around 00:57 UTC. The local ~/.hydracluster/config.yaml was updated, but a session holding the old value got a bare {"error":"unauthorized"} on every call including the exec transport, with nothing to indicate a rotation had happened.
A rotation of the registry token breaks THREE stores and none of them updates itself:
- the SCALE_REGISTRY_TOKEN secret on every repo that publishes an OCI image
- /etc/hydraskin/registry-auth.json on every hydraskin node (pull credential)
- /run/containers/0/auth.json on every hydraskin node, which the drop-in only re-copies on incus START, so editing the etc file is not enough and the staleness is invisible until the next pull
REFRESHED SO FAR (everything else is presumed stale)
- hydraissue repo secret, and both auth files on pi-node-004-nvme (2026-08-28)
- hydraorganization repo secret (2026-08-31)
PRESUMED STALE: every other repo publishing an image (hydravenues, hydraexperiencelibrary, hydranps, hydramirror, hydratransfer, hydrastreamingmonitor, hydranorthstar, hydrabodystatus, rogue and the pipeline repos) and every other hydraskin node.
ASK
- A documented, runnable fleet refresh: given a new registry token, update every repo secret and every node's two auth files, and report what it touched. The per-store procedure is already written up in hydraskin docs/runbooks/hydraskin.md under "Rotating the registry token"; what is missing is something that performs it across the fleet rather than one target at a time.
- A staleness check that can be run at any time: for each node, verify /run/containers/0/auth.json authenticates against the registry (curl -H "Authorization: Basic $AUTH" https://scaleregistry.experiencenet.com/v2/ expecting 200), and report any node that returns 401.
- Consider making rotation announce itself, so a caller gets something more useful than an unqualified 401.
ACCEPTANCE
- Rotating the registry token and running one documented procedure leaves zero repos and zero nodes on the old value, proven by the staleness check returning 200 everywhere and by a test release publishing successfully.
- The check can be run without rotating anything.