The registry now holds the deploy artifacts for six services and is a hard dependency for launching any scale. It has outgrown the operational surface it shipped with. State verified 2026-08-02.
Architecture (worth writing down — it was not obvious):
:443 hydrascaleregistry.service auth proxy, TLS termination, autocert
└─ 127.0.0.1:5000 docker-registry.service Distribution, config at /etc/docker/registry/config.yml
Holding: hydranps, hydravenues, hydraorganization, hydrapipeline, hydraexperiencelibrary, hydradistrict — 68 MB, 6% of a 38 GB disk.
hydrascaleregistry logs only startup lines and TLS handshake errors. There is no request logging at all.
This is already blocking a real investigation: pulls through incusd fail unauthorized while the identical skopeo command succeeds by hand (hydraskin #426), and the registry logs were silent throughout — which proves nothing, and I initially over-read as "the request never arrived".
Log method, path, source IP, and auth outcome. The distinction that matters: a request arriving with no Authorization header (client stripping it) versus arriving with credentials that were rejected. Those point at completely different fixes, and today we cannot tell them apart.
storage.delete.enabled: true is set, so manifests can be deleted — but nothing ever runs registry garbage-collect, so deleting a manifest never frees its blobs. Storage grows monotonically with every tag pushed.
maintenance.uploadpurging is enabled (168h), which handles abandoned uploads but not unreferenced blobs.
At 68 MB this is not urgent. It compounds silently, and the precedent is unpleasant: mirror-a reached 98% disk with a max_storage guard configured above the physical disk size, so it could never fire (#414).
Note GC in Distribution needs the registry read-only or stopped to be safe against concurrent pushes — worth scheduling deliberately rather than as a naive cron.
hydrabackup is not installed on this host, and it would not help if it were: it collects *.yaml service state and deliberately skips secrets, so registry blobs are outside its model entirely (see hydrabackup #425 for the boundary).
So the images are currently single-copy. They are rebuildable from source via CI, which softens this — but "rebuild six services from tags" is a bad recovery path during an incident, and CI has already proven able to produce phantom builds.
Decide explicitly: either back up /var/lib/registry, or write down that rebuild-from-CI is the recovery plan and confirm every image tag is reproducible.
Nothing watches disk on this host. Same class of problem as mirror-a. Cheap to add, and the failure mode without it is a registry that silently stops accepting pushes.
hydranps currently carries v0.2.3, v0.2.2, latest. Nothing prevents v0.2.3 being overwritten with different content, so a version tag is not a stable identifier. For a deploy artifact that is a meaningful weakness — a scale relaunched from v0.2.3 months later may not get what was tested.
Options: enforce immutability for v* tags in the auth proxy (reject PUT to an existing version tag), or pin by digest at launch time. The proxy already gates writes, so it is the natural place.
One host, no HA. Since no scale can be launched without it, a registry outage blocks all scale deployment and recovery. Acceptable today, but it should be a deliberate decision rather than an accident.
Related and already decided: the registry must not run as a scale — it cannot be the thing you need in order to recover the thing you need.
The planned edge mirroring (Distribution's proxy.remoteurl pull-through at district/venue) would also reduce this exposure, since an edge cache can serve a pull while the hub is down.