HydraIssues

What happened

brussels-district-v2 rebooted at 2026-08-03 07:17 UTC. wg-quick@wg0 was disabled, so the interface never came back. The district mesh — 24 peers — was down for roughly 11 hours before it was noticed, and only then because unrelated work needed to reach a venue.

wg-quick@wg0: inactive
enabled:      disabled
wg show wg0:  Unable to access interface: No such device
uptime:       up 11 hours, 21 minutes   (boot 2026-08-03 07:17)

Nothing alerted. hydraguard itself was running and logging error getting wireguard status: wg show: exit status 1 every 10 seconds for eleven hours, into a journal nobody was watching.

Fixed

wg-quick up wg0
systemctl enable wg-quick@wg0    # this is what was missing

The hub came back with the same public key, VGA6ETZB2XFVRRb5KmcFvQ+Ybfh9KKfcWuXfP1IuvQE=, all 24 peers configured, and handshakes resuming within seconds. /etc/wireguard/hub.key and wg0.conf were untouched throughout.

That is a useful confirmation of the restore drill on #422: the identity survived, and recovery needed no key material beyond what was already on disk.

Why this is worse than an eleven-hour outage

• The hub is a single point of failure for the whole district, and it now also fronts public ingress via hydrascalerouter. The same reboot would take customer-facing services down with it.
• The failure was silent. No alert, no dashboard signal. The one component that did know — hydraguard, logging every 10s — had nowhere to report it.
• It had presumably been disabled for a long time. The box last rebooted months ago, so nothing exercised the boot path. Any reboot at any point would have done this.

Worth doing

1. Audit every hydraguard host for systemctl is-enabled wg-quick@wg0. If the hub was disabled, peers may be too, and their failure would be equally silent.
2. Alert on hub reachability, not just on the service being up. hydraguard already detects this — it needs somewhere to send it. A peer count of zero, or wg show failing, should page.
3. Consider whether hydraguard should bring the interface up itself rather than relying on an external unit being enabled. It already owns wg0.conf; owning the interface lifecycle would remove this class of failure entirely.
4. Related: the district server is also the WireGuard hub and now the public ingress. Concentrating both on an instance whose backup is only recently proven (#422, #424) deserves a deliberate decision.

Note

The reboot cause is unconfirmed — most likely unattended-upgrades. Worth checking whether automatic reboots are enabled on this host, given what it carries.