HydraIssues

hydragon: automate the build so a submit to //hydragon/main produces a build with no human step
open feature Project: hydragon Parent: #615 Reporter: cederik 1 Sep 2026 07:00

Description

Sub-issue of #615. Make a submit to //hydragon/main produce a runnable build with nobody touching a keyboard.

What we have today

The pieces all exist. Nothing joins them.

  • hydraunrealengine cooks and packages through RunUAT, with preflight checks and a Perforce wrapper. It also has a serve mode with a local web interface and a job system (internal/web/job.go).
  • hydraperforcewatcher watches a depot path, zips what a changelist touched, uploads to hydramirror and notifies hydracluster.
  • hydracluster exec reaches the build machine, because it is a hydranode body.

So the missing piece is one trigger: a submit to the source path must start a package job on the build machine.

The loop we want

p4 submit source under //hydragon/main/<project>/
  -> trigger detects the changelist
  -> package job runs on the build machine (hydraunrealengine package)
  -> the package is submitted under //hydragon/main/Builds/
  -> hydraperforcewatcher publishes it to hydramirror and the library
  -> the body installs it

THE TRAP: this loop eats itself

Read this before writing any code.

The package is submitted back into the same depot. If the trigger watches the whole stream, that submit starts another cook, which submits another package, which starts another cook. An infinite build loop that fills the mirror and the depot pool in hours.

The two paths must be strictly separated:

  • //hydragon/main/<project>/... changes trigger a cook. Never a publish.
  • //hydragon/main/Builds/... changes trigger a publish. Never a cook.

Anything that matches both, or neither, is a bug. Test this before the first automated run, not after.

Design opinions, offered for argument

  1. Do not build on every submit. A cook takes tens of minutes and writes gigabytes. During active development somebody submits ten times an hour, and the queue never drains. Debounce: after a submit, wait for a quiet period, then build the newest changelist only. Never run two cooks at once, and never queue more than one pending build. Coalesce, do not accumulate.

  2. Retention has to land with this, not after. hydramirror has 2.52 GB free of 40 GB (#630). One automated build a day fills it in a week. Ship a retention policy in the same change: keep the last N builds, keep anything marked as a production build, delete the rest. Automation without retention is a slow outage.

  3. Development configuration by default. Cook Development, not Shipping, while we are developing. It is faster and it keeps symbols. Add a way to ask for Shipping explicitly when we want a real candidate.

  4. Release branches: not yet. Agreed. While the whole team is one lane, a release branch is ceremony with no payoff. Revisit when the first venue demo has a date, then a //hydragon/release/... stream builds Shipping and promotes automatically, and main stops promoting.

  5. Fail loudly. The two silent failures we already have in this chain (#609 stale token, #632 any 4xx) both mark work as processed and move on. Do not copy that pattern. A failed cook must be visible on the dashboard and must not mark the changelist done.

  6. Where the trigger lives. Two options. Either extend hydraperforcewatcher with a cook action for a watch, or write a small separate trigger. Extending the watcher is tempting because it already polls changelists, but it has a single global perforce.port (#631) and it currently means one thing by "watch". A separate small component with one job may age better. Decide with the code in front of you, not from this issue.

Scope

  • In: the trigger, the debounce, the cook job on the build machine, the submit of the package, the failure reporting, the retention policy.
  • Out: release branches, multi-platform cooks, incremental or distributed cooking, a build farm.

Done when

Somebody submits a change to the hydragon project source, walks away, and a new build appears in the experience library without anyone running a command. A broken cook shows up as a failure, not as silence. Two submits a minute apart produce one build, not two.

Depends on

  • #623, the depot must exist.
  • A build machine with hydraunrealengine installed.
  • #630, mirror capacity, or the first week of automation fills it.

Related: #615, #622, #631, #632, #609, #610.