HydraIssues

Issues / #630
hydramirror nbg1 has 2.52 GB free of 40 GB: no real Unreal package can upload
open bug Project: hydraperforce Reporter: cederik 31 Aug 2026 21:19

Description

Found during hydragon reconnaissance (#615) on 2026-08-31.

The nbg1 mirror is nearly full. Verified live: available_bytes 2522296320, used 35767435264. That is 2.52 GB free of 40 GB.

Why it matters

hydraperforcewatcher uploads every build archive to the mirror before it notifies hydracluster. The Gallo-Romeins packages are 5.3 GB each. A single real Unreal package will not fit in the space that is left.

hydragon Phase 1 (#623) submits a trivial proof artifact, which fits. Phase 2 (#624) submits a real packaged build, which does not. So this blocks Phase 2 and it will block the next Cyborn or Soulmade build of normal size too.

What to do

  1. Find what is holding the 35.8 GB. Old build archives are the likely answer.
  2. Decide a retention policy. Today nothing appears to expire.
  3. Free space, or grow the mirror volume.
  4. Add a free-space check to the watcher's pre-upload path, so a full mirror fails loudly instead of at the end of a long upload.

Related: #615, #624, #609 (the other silent failure in the same chain).

Comments (1)

api 18 Sep 2026 13:47

This bit production today (2026-09-18). Cyborn submitted gallo-romeins-museum CL9 (4.8 GB build.zip, "Automated deploy from Jenkins E") and every watcher upload attempt failed with HTTP 507 storage limit exceeded from nbg1.hydramirror; the watcher retried every 5 minutes since 13:09 UTC. The build never reached the mirror or the experience library.

Unblocked by deleting the stale builds/galloromeins/ tree on the nbg1 mirror (5.0 GB, 353 loose files from the 2026-08-14 standalone-box trial era, old watch name, referenced by nothing in the experience library, re-syncable from the Perforce depot). Deleted via the mirror DELETE API so files.yaml stays consistent. Store now 29.05 GB of the 32 GB max_storage; disk 6.3 GB free of 38. CL9 then uploaded and registered as library build 4 at 13:45 UTC.

Still fragile: max_storage 32GB with retention DISABLED in /var/lib/hydramirror/config.yaml, and gallo builds are ~5 GB each, so the next 1-2 Cyborn submits hit 507 again. builds/gallo-romeins-museum holds CL 1/5/8/9 (~16 GB). Options: enable retention (releases/ only, ~9.6 GB there) plus a builds/ retention policy, grow the disk, or point the watcher at bxl1 (~205 GB free).