HydraIssues

Found during hydragon reconnaissance (#615) on 2026-08-31. Same failure shape as #609, different cause.

The problem

When hydraperforcewatcher posts a build notification and gets any 4xx back, it logs a warning, marks the changelist processed, and moves on. There is no retry and no alert.

#609 was this failure with a stale authentication token, which returned 401. The same silent drop happens on 404, which is what you get when the experience does not exist in hydraexperiencelibrary, or when its watch name does not match the watcher's watch name.

Why it matters

The failure is invisible from every direction. The submit succeeded. The archive uploaded to the mirror. The watcher looks healthy on the dashboard. The changelist is marked processed, so it never retries. The only symptom is a build that never appears, and nobody notices until someone goes looking.

For hydragon this means the experience record must exist, with a matching watch name, before the first submit. That ordering requirement is invisible in the code and easy to get wrong.

What to do

1. Treat 4xx as a failure that leaves the changelist unprocessed, or record it in a dead letter list that something reports on.
2. Distinguish the causes. A 401 is a credentials problem, a 404 is a configuration problem, and they need different fixes.
3. Surface unprocessed changelists on the hydraperforce dashboard. Today the agent shows healthy while builds are being dropped.
4. Add a startup check: for every configured watch, confirm the matching experience exists before accepting any changelist.

Related: #609, #610, #615, #623.