HydraIssues

District WireGuard hub did not survive a reboot — wg-quick@wg0 was disabled, mesh down ~11h undetected
open bug Priority: critical Project: hydraguard Reporter: 3 Aug 2026 18:44

Description

## What happened

`brussels-district-v2` rebooted at **2026-08-03 07:17 UTC**. `wg-quick@wg0` was **`disabled`**, so the interface never came back. The district mesh — 24 peers — was down for roughly **11 hours** before it was noticed, and only then because unrelated work needed to reach a venue.

```
wg-quick@wg0: inactive
enabled: disabled
wg show wg0: Unable to access interface: No such device
uptime: up 11 hours, 21 minutes (boot 2026-08-03 07:17)
```

Nothing alerted. `hydraguard` itself was running and logging `error getting wireguard status: wg show: exit status 1` every 10 seconds for eleven hours, into a journal nobody was watching.

## Fixed

```bash
wg-quick up wg0
systemctl enable wg-quick@wg0 # this is what was missing
```

The hub came back with the **same public key**, `VGA6ETZB2XFVRRb5KmcFvQ+Ybfh9KKfcWuXfP1IuvQE=`, all 24 peers configured, and handshakes resuming within seconds. `/etc/wireguard/hub.key` and `wg0.conf` were untouched throughout.

That is a useful confirmation of the restore drill on #422: the identity survived, and recovery needed no key material beyond what was already on disk.

## Why this is worse than an eleven-hour outage

- **The hub is a single point of failure for the whole district**, and it now also fronts public ingress via hydrascalerouter. The same reboot would take customer-facing services down with it.
- **The failure was silent.** No alert, no dashboard signal. The one component that *did* know — hydraguard, logging every 10s — had nowhere to report it.
- **It had presumably been disabled for a long time.** The box last rebooted months ago, so nothing exercised the boot path. Any reboot at any point would have done this.

## Worth doing

1. **Audit every hydraguard host for `systemctl is-enabled wg-quick@wg0`.** If the hub was disabled, peers may be too, and their failure would be equally silent.
2. **Alert on hub reachability**, not just on the service being up. `hydraguard` already detects this — it needs somewhere to send it. A peer count of zero, or `wg show` failing, should page.
3. **Consider whether hydraguard should bring the interface up itself** rather than relying on an external unit being enabled. It already owns `wg0.conf`; owning the interface lifecycle would remove this class of failure entirely.
4. Related: the district server is also the WireGuard hub *and* now the public ingress. Concentrating both on an instance whose backup is only recently proven (#422, #424) deserves a deliberate decision.

## Note

The reboot cause is unconfirmed — most likely unattended-upgrades. Worth checking whether automatic reboots are enabled on this host, given what it carries.