HydraIssues

The gallo-romeins watcher on hydraskin-perforce-1 (178.105.185.28) held a stale hydracluster.admin_token. Every build notification POST to https://hydracluster.experiencenet.com/api/v1/builds/notify returned HTTP 401.

Impact: Cyborn submitted CL8 on 2026-08-31 11:53. The watcher synced and uploaded it to the mirror correctly (builds/gallo-romeins-museum/8/build.zip), then failed to register it in the experience library. The build was invisible to the platform until replayed by hand. CL5 on 2026-08-14 most likely hit the same failure.

Two design problems make this silent:

1. The notification failure is logged at WARN and treated as non-fatal (pkg/watcher/watcher.go:220). The changelist is then marked processed, so the build is never retried.
2. Nothing surfaces a failing agent. The hydraperforce dashboard shows the agent as healthy because state pushes succeed independently of notification failures.

Fixed on the node 2026-08-31 by writing the current hydracluster admin token into /root/.hydraperforcewatcher/config.yaml (backup kept alongside) and restarting the service. CL8 was then replayed by hand and is registered as build #3.

Asks:

• Do not mark a changelist processed when the notification fails, or record it as needing retry so the next poll retries it.
• Treat 401/403 as a distinct fatal-ish condition: surface it in the agent state push so the dashboard shows the agent as degraded.
• Audit the other watcher agent for the same stale token.