Description
## Context
The registry now holds the deploy artifacts for six services and is a **hard dependency for launching any scale**. It has outgrown the operational surface it shipped with. State verified 2026-08-02.
**Architecture** (worth writing down — it was not obvious):
```
:443 hydrascaleregistry.service auth proxy, TLS termination, autocert
└─ 127.0.0.1:5000 docker-registry.service Distribution, config at /etc/docker/registry/config.yml
```
Holding: `hydranps`, `hydravenues`, `hydraorganization`, `hydrapipeline`, `hydraexperiencelibrary`, `hydradistrict` — 68 MB, 6% of a 38 GB disk.
## 1. No per-request logging (do this first)
`hydrascaleregistry` logs only startup lines and TLS handshake errors. There is **no request logging at all**.
This is already blocking a real investigation: pulls through incusd fail `unauthorized` while the identical skopeo command succeeds by hand (hydraskin #426), and the registry logs were silent throughout — which proves nothing, and I initially over-read as "the request never arrived".
Log method, path, source IP, and **auth outcome**. The distinction that matters: a request arriving with *no* `Authorization` header (client stripping it) versus arriving with credentials that were *rejected*. Those point at completely different fixes, and today we cannot tell them apart.
## 2. No garbage collection
`storage.delete.enabled: true` is set, so manifests *can* be deleted — but **nothing ever runs `registry garbage-collect`**, so deleting a manifest never frees its blobs. Storage grows monotonically with every tag pushed.
`maintenance.uploadpurging` is enabled (168h), which handles abandoned uploads but not unreferenced blobs.
At 68 MB this is not urgent. It compounds silently, and the precedent is unpleasant: mirror-a reached 98% disk with a `max_storage` guard configured above the physical disk size, so it could never fire (#414).
Note GC in Distribution needs the registry read-only or stopped to be safe against concurrent pushes — worth scheduling deliberately rather than as a naive cron.
## 3. No backup
`hydrabackup` is not installed on this host, and it would not help if it were: it collects `*.yaml` service state and deliberately skips secrets, so registry *blobs* are outside its model entirely (see hydrabackup #425 for the boundary).
So the images are currently single-copy. They are rebuildable from source via CI, which softens this — but "rebuild six services from tags" is a bad recovery path during an incident, and CI has already proven able to produce phantom builds.
Decide explicitly: either back up `/var/lib/registry`, or write down that rebuild-from-CI *is* the recovery plan and confirm every image tag is reproducible.
## 4. No storage alerting
Nothing watches disk on this host. Same class of problem as mirror-a. Cheap to add, and the failure mode without it is a registry that silently stops accepting pushes.
## 5. Mutable tags
`hydranps` currently carries `v0.2.3`, `v0.2.2`, `latest`. Nothing prevents `v0.2.3` being overwritten with different content, so a version tag is not a stable identifier. For a deploy artifact that is a meaningful weakness — a scale relaunched from `v0.2.3` months later may not get what was tested.
Options: enforce immutability for `v*` tags in the auth proxy (reject `PUT` to an existing version tag), or pin by digest at launch time. The proxy already gates writes, so it is the natural place.
## 6. Single instance, now on the critical path
One host, no HA. Since no scale can be launched without it, a registry outage blocks all scale deployment and recovery. Acceptable today, but it should be a deliberate decision rather than an accident.
Related and already decided: the registry **must not run as a scale** — it cannot be the thing you need in order to recover the thing you need.
The planned edge mirroring (Distribution's `proxy.remoteurl` pull-through at district/venue) would also reduce this exposure, since an edge cache can serve a pull while the hub is down.
## Suggested order
1. Request logging — cheap, and unblocks hydraskin #426
2. Storage alerting — cheap, prevents a silent-fill incident
3. Garbage collection — scheduled with the read-only caveat understood
4. Backup decision — either implement, or document rebuild-from-CI as the plan
5. Tag immutability
6. Edge mirroring / HA posture