Description
## Summary
mirror-a is **the only copy of every artifact published since 2026-02-23** and has no peer
to fall back on. Found while diagnosing #407; unrelated to that ticket and more dangerous
than it was.
The disk half is fixed. **The redundancy half is not.**
## Fixed already
mirror-a was at **98% disk — 898 MB free of 38 GB** — and had already hit
`no space left on device` on 2026-07-24 while writing an upload.
Root cause of the pressure was unbounded journald (3.6 GB, no `SystemMaxUse`) plus 297 MB
of `btmp` brute-force login records. Both reclaimed; `SystemMaxUse=400M` now set so it
stays bounded. Free space went **898 MB → 4.6 GB** (98% → 88%) without touching any mirror
data.
Separately, `max_storage` was configured as **50 GB on a 38 GB disk**, so the guard in
`HandlePutFile` could never fire — writes were accepted until the filesystem physically
filled, which is exactly what happened in July. Now set to **34 GB**, which sits above the
31.2 GB currently tracked (so publishing keeps working) but below the disk, so the guard
actually engages. Verified after the change: PUT returns 201, GET returns 200.
## NOT fixed — no redundancy
`peers: static: []` and no `cluster_url`, so `peers: 0`. Pull-through can never satisfy a
miss, which means a single missing file is terminal.
Worse, the origin (`/var/www/releases`) **stopped receiving artifacts at the Feb 23 mirror
cutover**. It still holds a stale partial copy frozen at that date. So for everything
published in the last five months, mirror-a is the sole copy — one 38 GB disk, no backup,
no peer, 2,994 files / 30 GB.
Losing that host loses every release artifact since February.
## Recommended
1. **Restore a second copy.** Either configure a peer / `cluster_url` so pull-through can
recover a miss, or put the 30 GB under backup. This is the important one.
2. **Grow the disk.** 4.6 GB free on a store that only grows is a short runway; the guard
now stops writes cleanly rather than corrupting, but stopping writes still breaks
publishing.
3. **Fix the origin divergence.** `hydrarelease`'s own workflow still rsyncs to
`deploy@releases.experiencenet.com:/var/www/releases/hydrarelease` — files that are
written but never served. Either drop that block or reinstate the origin as a real
second copy (which would also address point 1).
4. **Bound the anonymous write amplification.** `HandleGetFile` writes a permanent
`status: failed` entry to `files.yaml` on every unauthenticated miss — 158 entries so
far, including vulnerability-scanner noise (`vendor/laravel-filemanager/js/script.js`,
`error_log.php`). Only persist stubs for authenticated or peer-backed paths.
## Also worth noting
`btmp` reaching 297 MB indicates sustained SSH brute-force traffic against this host. Not
urgent, but a signal worth looking at.
`mirror-a` is no longer the sole copy. There is now an hourly, checksum-verified offsite copy of the whole artifact store on a Hetzner Storage Box.
**Nothing under `/var/lib/hydramirror` was deleted, moved, truncated or overwritten.** The sync is read-only at the source and append-only at the destination. `hydramirror` was never stopped or restarted; free space on `mirror-a` was 3.5 GB before and 3.5 GB after.
### Why a Storage Box and not a peer
I read the peering code before assuming it would help. It would not have.
`peer.Client.Pull` only fires from `HandleGetFile` on a local miss — pull-through is **on demand only**. There is no background replication anywhere in the codebase. A new peer would have started empty and stayed empty except for files somebody happened to request. It would have satisfied `peers: 0` in the health output without ever being a second copy — arguably worse than the current state, because it would have *looked* fixed.
Restoring origin sync from `hydrarelease` was also not it: that workflow only rsyncs **hydrarelease's own** artifacts to `/var/www/releases/hydrarelease`, so reinstating it recovers one project out of ~45.
I measured the origin rather than trusting either description of it. `/var/www/releases` on `46.225.120.7` holds 19 GB across 29 projects — so it is **not** completely frozen at Feb 23 as this ticket states; `land v0.87.0` landed there today, and `hydrarelease` as recently as 2026-07-20. But it has diverged so far that it was never a usable fallback:
```
mirror-a ready files : 3123 30.08 GiB
also present on origin : 42 0.37 GiB
SOLE COPY on mirror-a : 3081 29.71 GiB (98.8%)
```
So the ticket's conclusion was right even though the mechanism was slightly off: a handful of projects still write to the origin, but **98.8% of the store existed in exactly one place.** That is the number this work removes.
### What was actually done
| | |
|---|---|
| Destination | Storage Box `storage-box-1` / subaccount `u520179-sub2`, `/home/mirror-a` |
| Sync | `/usr/local/bin/hydramirror-offsite-backup.sh` via `hydramirror-offsite.timer` (hourly) |
| Verify | `/usr/local/bin/hydramirror-offsite-verify.sh` |
| Log | `/var/log/hydramirror-offsite.log` |
| Auth | dedicated SSH key `/root/.ssh/hydramirror_backup`, port 23 |
Deliberate choices:
- **No `--delete`.** The destination is append-only, so data loss on `mirror-a` can never propagate to the backup.
- **`-H`** preserves the hardlinks made by `POST /api/v1/link`.
- **`config.yaml` is not copied verbatim** — it goes as `config.yaml.redacted` with `admin_token` stripped. The artifacts are irreplaceable; the token is not, and an unencrypted offsite copy of it only widens the blast radius.
- **`flock`** so an hourly run can never overlap a long one.
Also hardened, both reversible: daily Storage Box snapshots (04:30 UTC, 10 retained), delete-protection on the Storage Box, and **delete + rebuild protection on `mirror-a`**. Rebuild protection deliberately makes a host-snapshot restore a two-step act, so nobody accidentally rolls the host back and reverts every artifact published since the snapshot.
### Verification — not "rsync exited 0"
This is the part the ticket rightly cares about. `files.yaml` already stores a SHA256 per file from ingest, so I re-hashed **every file on the Storage Box** and compared against the digest hydramirror itself computed:
```
expecting 3122 files present on mirror-a
----------------------------------------
verified OK : 3122 / 3122
MISMATCHED : 0
MISSING : 0
store entries with no file on mirror-a (pre-existing, not a backup fault): 1
--- orphaned store entries ---
guardcheck/probe.txt
----------------------------------------
RESULT: offsite copy is complete and byte-identical to the recorded hashes
```
Because the comparison is against ingest-time hashes, this also independently proves there is **no bit-rot on `mirror-a`** — all 3122 files still hash to what they hashed to when they were published.
Two incidental findings while checking consistency:
- Every file on disk is tracked, and every tracked file exists — **except `guardcheck/probe.txt`**, a `ready` entry in `files.yaml` whose file is gone. It is a health probe, not an artifact, so nothing irreplaceable is lost, but the store is carrying one stale row. The verify script reports these separately rather than failing on them.
- The Storage Box reports 23 GiB for a 30 GiB payload. That is ZFS compression, not a short copy — the hash pass above is the authority.
### Correction to this ticket's premise
Hetzner **automated daily host backups were already enabled** on `mirror-a` (window 10:00–14:00 UTC, 7 retained, newest 2026-08-03 10:21). So "no backup" was not strictly accurate. But it was never verified, and it is the wrong instrument for this risk: a whole-host image can only be restored by rolling the server back, which reverts every artifact published since the snapshot. The tool for recovering a lost file was also a tool for losing newer ones. It is now a genuine second layer behind a file-level copy, rather than the only layer.
The runbook previously asserted these snapshots as "what is backed up" with an unexercised restore procedure. Corrected in cederikdotcom/hydramirror#1.
---
## Capacity — analysis and proposal (nothing executed)
### The guard still does not protect what it was meant to protect
From the live health endpoint:
```
total_size 32,300,752,346 (tracked)
max_storage 36,507,222,016 (34GB, and parseSize treats GB as 2^30)
disk available 3,740,553,216
```
Guard headroom is **3.92 GiB** but real free space is **3.48 GiB**. The filesystem still fills ~450 MB *before* the guard can fire, so the July failure mode — `ENOSPC` mid-upload instead of a clean `507` — is still reachable. Lowering `max_storage` from `34GB` to `32GB` closes that (still comfortably above the 30.08 GiB tracked, so publishing is unaffected today). I did not change it, because it also shortens the runway to a hard publish stop and that is an availability call to make alongside the disk decision below.
### Runway
August is running ~258 MB/day (0.72 GiB in 3 days), consistent with the ~8.1 GiB/month average over the last three full months. At that rate **`mirror-a` runs out of physical disk in roughly two weeks.** `hydrascalerouter` shipping ~108 MB per release across two architectures makes this worse, not better.
### What is actually consuming 30 GiB
27.31 GiB of the 30.08 GiB is `releases/`, and 27.23 GiB of that is the `production` channel. It is almost entirely superseded versions:
| Retention | Retained | Reclaimable |
|---|---|---|
| keep newest 1 | 1.82 GiB | 25.49 GiB |
| keep newest 3 | 4.99 GiB | 22.32 GiB |
| **keep newest 5** | **7.87 GiB** | **19.44 GiB** |
Worst offenders under keep-newest-5: `hydrahead` (75 versions, 3.65 GiB reclaimable), `hydrabody` (53, 2.93), `hydraexperiencenet` (47, 2.56), `hydraapplepipeline` (40, 1.97), `hydracluster` (76 versions for 1.08).
**Proposal — keep the newest 5 versions per project+channel.** That takes the store from 30.08 GiB to ~10.6 GiB and turns a two-week runway into roughly two years at current growth. I have **not** deleted anything and am not proposing that anyone do so by hand: `hydramirror` has no retention code today (`store.CleanupFailed` only touches `failed`/`pulling` stubs), so this needs a real `prune` subcommand with `--dry-run`, tests, and a tag — and it should only run now that a verified second copy exists and can be re-verified afterwards.
Alternatives if deleting released artifacts is unacceptable:
- **`cax11` → `cax21`** — 40 GB to 80 GB disk, **+4.50 EUR/mo**. Needs a power-off, and Hetzner disk growth is irreversible (no downgrade afterwards).
- **100 GB Hetzner Volume** — **~4.40 EUR/mo**, hot-attaches with no reboot, but only helps once `data_dir` moves onto it, which means relocating the only copy. Safer to attempt now that the offsite copy is verified, but still the riskiest of the three.
My recommendation: retention first (free, largest effect, and the Storage Box already holds everything being pruned), disk growth only if the policy is rejected.
## Also proposed, not done
- **Bound the anonymous write amplification** (item 4). `HandleGetFile` still persists a `status: failed` row on every unauthenticated miss — now 163 entries, including scanner noise. Unauthenticated misses should not be able to grow `files.yaml` without limit.
- **Drop the dead `/var/www/releases` rsync** from `hydrarelease`'s `release.yml` (lines 77–94). It writes files nobody serves and creates a stale partial copy that reads like a backup — exactly the confusion that made this ticket necessary.
## Cost
**Zero incremental.** The Storage Box (`storage-box-1`, bx11, 1 TiB, 3.20 EUR/mo) already existed and was completely empty — 0 bytes used since it was created in December 2025. The offsite copy uses 23 GiB of 1 TiB, so there is room for retention offload later. No server was created. The proposals above are the only things that would add cost.